Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 881647 Details for
Bug 1082365
attribute error when executing fenceSpmStorage
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
log when Node1 was powerded off
engine-node1-down.log (text/plain), 167.18 KB, created by
md
on 2014-04-02 07:09:09 UTC
(
hide
)
Description:
log when Node1 was powerded off
Filename:
MIME Type:
Creator:
md
Created:
2014-04-02 07:09:09 UTC
Size:
167.18 KB
patch
obsolete
>2014-04-02 08:46:47,824 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-04-02 08:46:47,825 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-04-02 08:46:47,826 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-04-02 08:46:47,827 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-04-02 08:46:47,829 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-04-02 08:46:47,830 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-04-02 08:46:47,831 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Loaded file "/etc/ovirt-engine/engine.conf.d/20-ovirt-engine-reports.conf". >2014-04-02 08:46:47,832 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-04-02 08:46:47,832 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_AJP_PORT" is "8702". >2014-04-02 08:46:47,833 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_APPS" is "engine.ear "/var/lib/ovirt-engine-reports/ovirt-engine-reports.war"". >2014-04-02 08:46:47,834 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-04-02 08:46:47,835 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-04-02 08:46:47,836 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-04-02 08:46:47,836 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-04-02 08:46:47,837 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-04-02 08:46:47,838 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_HOST" is "localhost". >2014-04-02 08:46:47,839 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-04-02 08:46:47,839 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-04-02 08:46:47,840 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-04-02 08:46:47,841 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_PORT" is "5432". >2014-04-02 08:46:47,841 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_SECURED" is "False". >2014-04-02 08:46:47,842 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-04-02 08:46:47,843 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-04-02 08:46:47,844 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DB_USER" is "engine". >2014-04-02 08:46:47,844 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-04-02 08:46:47,845 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-04-02 08:46:47,846 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-04-02 08:46:47,846 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_FQDN" is "engine.ws-datenservice.de". >2014-04-02 08:46:47,847 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_GROUP" is "ovirt". >2014-04-02 08:46:47,848 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-04-02 08:46:47,849 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-04-02 08:46:47,849 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-04-02 08:46:47,850 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-04-02 08:46:47,851 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-04-02 08:46:47,851 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-04-02 08:46:47,852 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_HTTP_PORT" is "None". >2014-04-02 08:46:47,853 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules:/var/lib/ovirt-engine-reports/modules". >2014-04-02 08:46:47,853 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-04-02 08:46:47,854 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-04-02 08:46:47,855 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-04-02 08:46:47,856 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-04-02 08:46:47,856 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PERM_MAX" is "256m". >2014-04-02 08:46:47,857 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PERM_MIN" is "256m". >2014-04-02 08:46:47,858 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-04-02 08:46:47,859 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-04-02 08:46:47,859 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-04-02 08:46:47,860 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-04-02 08:46:47,861 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-04-02 08:46:47,862 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-04-02 08:46:47,863 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-04-02 08:46:47,863 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-04-02 08:46:47,864 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false". >2014-04-02 08:46:47,865 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-04-02 08:46:47,866 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-04-02 08:46:47,866 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-04-02 08:46:47,867 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-04-02 08:46:47,868 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-04-02 08:46:47,869 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_STOP_TIME" is "10". >2014-04-02 08:46:47,869 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-04-02 08:46:47,870 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-04-02 08:46:47,871 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-04-02 08:46:47,872 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_USER" is "ovirt". >2014-04-02 08:46:47,873 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-04-02 08:46:47,873 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-04-02 08:46:47,874 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-04-02 08:46:47,875 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "JBOSS_HOME" is "/usr/share/jboss-as". >2014-04-02 08:46:47,875 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-12) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-04-02 08:46:48,326 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-16) Start initializing Backend >2014-04-02 08:46:48,674 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-16) Running ovirt-engine 3.4.0-1.el6 >2014-04-02 08:46:48,676 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-16) Start initializing dictionaries >2014-04-02 08:46:48,682 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-16) Finished initializing dictionaries >2014-04-02 08:46:48,683 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-16) Start initializing AuditLogCleanupManager >2014-04-02 08:46:48,685 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-16) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-04-02 08:46:48,711 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-16) Finished initializing AuditLogCleanupManager >2014-04-02 08:46:48,716 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-16) Start initializing TagsDirector >2014-04-02 08:46:48,730 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-16) Tag root added to tree >2014-04-02 08:46:48,745 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-16) Finished initializing TagsDirector >2014-04-02 08:46:48,749 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-16) Start initializing IsoDomainListSyncronizer >2014-04-02 08:46:48,757 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-16) Finished initializing IsoDomainListSyncronizer >2014-04-02 08:46:48,769 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-16) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-04-02 08:46:48,999 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-16) Completed initializing handlers >2014-04-02 08:46:49,001 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-16) Start initializing ErrorTranslatorImpl >2014-04-02 08:46:49,016 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-16) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-04-02 08:46:49,018 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-16) Finished initializing ErrorTranslatorImpl >2014-04-02 08:46:49,019 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-16) Start initializing ErrorTranslatorImpl >2014-04-02 08:46:49,020 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-16) Finished initializing ErrorTranslatorImpl >2014-04-02 08:46:49,022 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-16) Mark incomplete jobs as UNKNOWN >2014-04-02 08:46:49,060 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-16) Start initializing JobRepositoryCleanupManager >2014-04-02 08:46:49,062 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-16) Finished initializing JobRepositoryCleanupManager >2014-04-02 08:46:49,064 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-16) Start initializing AutoRecoveryManager >2014-04-02 08:46:49,065 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-16) Finished initializing AutoRecoveryManager >2014-04-02 08:46:49,068 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-16) Start initializing ExecutionMessageDirector >2014-04-02 08:46:49,070 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-16) Finished initializing ExecutionMessageDirector >2014-04-02 08:46:49,130 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-13) Start initializing UsersDomainsCacheManagerService >2014-04-02 08:46:49,140 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-13) Start initializing DbUserCacheManager >2014-04-02 08:46:49,143 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-13) Finished initializing DbUserCacheManager >2014-04-02 08:46:49,144 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-13) Finished initializing UsersDomainsCacheManagerService >2014-04-02 08:46:49,155 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (MSC service thread 1-15) Initialization of AsyncTaskManager completed successfully. >2014-04-02 08:46:49,161 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) Start initializing ResourceManager >2014-04-02 08:46:49,410 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Entered VdsManager constructor >2014-04-02 08:46:49,445 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Initialize vdsBroker (192.168.71.23,54,321) >2014-04-02 08:46:49,584 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) VDS 80910869-bf75-4485-9fe6-f1c73a1e667d was added to the Resource Manager >2014-04-02 08:46:49,585 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Entered VdsManager constructor >2014-04-02 08:46:49,600 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Initialize vdsBroker (192.168.71.22,54,321) >2014-04-02 08:46:49,606 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) VDS ff474b41-22c5-440e-8052-4cf40c27b250 was added to the Resource Manager >2014-04-02 08:46:49,641 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) Finished initializing ResourceManager >2014-04-02 08:46:49,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-15) Initialization of OvfDataUpdater completed successfully. >2014-04-02 08:46:49,655 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Start scheduling to enable vds load balancer >2014-04-02 08:46:49,657 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Finished scheduling to enable vds load balancer >2014-04-02 08:46:49,658 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Start HA Reservation check >2014-04-02 08:46:49,660 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Finished HA Reservation check >2014-04-02 08:46:49,671 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (org.ovirt.thread.pool-6-thread-1) MacPoolManager(36a69dd): Start initializing >2014-04-02 08:46:49,686 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-15) Init VM custom properties utilities >2014-04-02 08:46:49,688 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-15) Init device custom properties utilities >2014-04-02 08:46:49,700 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (org.ovirt.thread.pool-6-thread-1) MacPoolManager(36a69dd): Finished initializing. Available MACs in pool: 253 >2014-04-02 08:46:49,701 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Initializing Scheduling manager >2014-04-02 08:46:49,742 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) External scheduler disabled, discovery skipped >2014-04-02 08:46:49,743 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Initialized Scheduling manager >2014-04-02 08:46:49,744 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-15) Initializing DWH Heart Beat >2014-04-02 08:46:49,746 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-15) DWH Heart Beat initialized >2014-04-02 08:46:50,810 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [3edcfdc8] Running command: LogoutUserCommand internal: false. >2014-04-02 08:46:50,869 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: UNASSIGNED not exist in string table >2014-04-02 08:46:50,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-04-02 08:46:50,872 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-04-02 08:46:50,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-04-02 08:46:50,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-04-02 08:46:50,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: DWH_STOPPED not exist in string table >2014-04-02 08:46:50,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: DWH_STARTED not exist in string table >2014-04-02 08:46:50,879 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: DWH_ERROR not exist in string table >2014-04-02 08:46:50,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_REMOVE_VM not have severity. Assumed Normal >2014-04-02 08:46:50,893 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2014-04-02 08:46:50,894 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-02 08:46:50,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-02 08:46:50,896 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2014-04-02 08:46:50,898 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI not have severity. Assumed Normal >2014-04-02 08:46:50,899 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: GLUSTER_SERVICES_LIST_NOT_FETCHED not have severity. Assumed Normal >2014-04-02 08:46:50,900 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: GLUSTER_VOLUME_BRICK_ADDED not have severity. Assumed Normal >2014-04-02 08:46:50,901 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_EXTEND_DISK_SIZE_UPDATE_VM_FAILURE not have severity. Assumed Normal >2014-04-02 08:46:50,902 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2014-04-02 08:46:50,903 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2014-04-02 08:46:50,905 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2014-04-02 08:46:50,906 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2014-04-02 08:46:50,907 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_ADDED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,908 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_TO_ADD_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,909 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_UPDATED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,910 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_TO_UPDATE_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,911 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_REMOVED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,912 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] AuditLogType: USER_FAILED_TO_REMOVE_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:46:50,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3edcfdc8] Correlation ID: 3edcfdc8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-04-02 08:46:53,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vds=Host[Node2]), log id: 72ea1931 >2014-04-02 08:46:53,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]), log id: 6d0a720 >2014-04-02 08:46:53,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 72ea1931 >2014-04-02 08:46:53,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 6d0a720 >2014-04-02 08:46:53,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host Node2. >2014-04-02 08:46:53,368 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host Node1. >2014-04-02 08:46:53,459 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: Node2 >2014-04-02 08:46:53,531 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [65cdff1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 80910869-bf75-4485-9fe6-f1c73a1e667d Type: VDS >2014-04-02 08:46:53,562 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: Node1 >2014-04-02 08:46:53,594 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [1d947bf0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:46:59,661 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000000ea Type: StoragePool >2014-04-02 08:46:59,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [7d14527f] Correlation ID: 7d14527f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-04-02 08:47:04,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] hostFromVds::selectedVds - Node2, spmStatus Free, storage pool Default >2014-04-02 08:47:04,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] Found spm host ff474b41-22c5-440e-8052-4cf40c27b250, host name: Node1, according to spmId: 1. >2014-04-02 08:47:07,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] SpmStatus on vds ff474b41-22c5-440e-8052-4cf40c27b250: SPM >2014-04-02 08:47:07,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] Using old spm server: Node1, no start needed >2014-04-02 08:47:07,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [7d14527f] Initialize Irs proxy from vds: 192.168.71.22 >2014-04-02 08:47:07,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [7d14527f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host Node1 (Address: 192.168.71.22). >2014-04-02 08:47:07,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-24) [7d14527f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false), log id: 53304404 >2014-04-02 08:47:07,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-24) [7d14527f] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000000ea >2014-04-02 08:47:07,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-24) [7d14527f] START, HSMGetAllTasksInfoVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250), log id: 4379057 >2014-04-02 08:47:07,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-24) [7d14527f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4379057 >2014-04-02 08:47:07,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-24) [7d14527f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 53304404 >2014-04-02 08:47:07,873 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-24) [7d14527f] Discovered no tasks on Storage Pool Default >2014-04-02 08:47:18,550 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginAdminUserCommand internal: false. >2014-04-02 08:47:18,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-04-02 08:47:19,413 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-04-02 08:47:19,467 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (ajp--127.0.0.1-8702-7) Can't read file "/var/lib/ovirt-engine/reports.xml" for request "/ovirt-engine/services/reports-ui", will send a 404 error response. >2014-04-02 08:47:49,738 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-38) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:47:49,741 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-38) There is no over-utilized host in cluster BCE >2014-04-02 08:48:49,815 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-78) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:48:49,816 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-78) There is no over-utilized host in cluster BCE >2014-04-02 08:49:49,892 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-22) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:49:49,893 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-22) There is no over-utilized host in cluster BCE >2014-04-02 08:49:20,870 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-04-02 08:49:20,871 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-04-02 08:49:20,872 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-04-02 08:49:20,875 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-04-02 08:49:20,877 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-04-02 08:49:20,878 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-04-02 08:49:20,880 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Loaded file "/etc/ovirt-engine/engine.conf.d/20-ovirt-engine-reports.conf". >2014-04-02 08:49:20,882 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-04-02 08:49:20,883 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_AJP_PORT" is "8702". >2014-04-02 08:49:20,885 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_APPS" is "engine.ear "/var/lib/ovirt-engine-reports/ovirt-engine-reports.war"". >2014-04-02 08:49:20,886 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-04-02 08:49:20,888 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-04-02 08:49:20,889 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-04-02 08:49:20,891 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-04-02 08:49:20,892 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-04-02 08:49:20,893 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_HOST" is "localhost". >2014-04-02 08:49:20,894 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-04-02 08:49:20,895 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-04-02 08:49:20,896 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-04-02 08:49:20,898 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_PORT" is "5432". >2014-04-02 08:49:20,899 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_SECURED" is "False". >2014-04-02 08:49:20,900 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-04-02 08:49:20,901 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-04-02 08:49:20,903 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DB_USER" is "engine". >2014-04-02 08:49:20,904 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-04-02 08:49:20,905 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-04-02 08:49:20,906 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-04-02 08:49:20,907 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_FQDN" is "engine.ws-datenservice.de". >2014-04-02 08:49:20,908 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_GROUP" is "ovirt". >2014-04-02 08:49:20,909 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-04-02 08:49:20,910 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-04-02 08:49:20,912 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-04-02 08:49:20,913 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-04-02 08:49:20,914 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-04-02 08:49:20,915 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-04-02 08:49:20,916 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_HTTP_PORT" is "None". >2014-04-02 08:49:20,917 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules:/var/lib/ovirt-engine-reports/modules". >2014-04-02 08:49:20,918 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-04-02 08:49:20,920 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-04-02 08:49:20,921 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-04-02 08:49:20,922 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-04-02 08:49:20,923 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PERM_MAX" is "256m". >2014-04-02 08:49:20,924 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PERM_MIN" is "256m". >2014-04-02 08:49:20,925 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-04-02 08:49:20,926 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-04-02 08:49:20,927 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-04-02 08:49:20,929 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-04-02 08:49:20,930 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-04-02 08:49:20,931 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-04-02 08:49:20,932 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-04-02 08:49:20,933 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-04-02 08:49:20,933 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false". >2014-04-02 08:49:20,934 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-04-02 08:49:20,935 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-04-02 08:49:20,936 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-04-02 08:49:20,937 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-04-02 08:49:20,937 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-04-02 08:49:20,938 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_STOP_TIME" is "10". >2014-04-02 08:49:20,939 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-04-02 08:49:20,940 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-04-02 08:49:20,940 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-04-02 08:49:20,941 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_USER" is "ovirt". >2014-04-02 08:49:20,942 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-04-02 08:49:20,943 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-04-02 08:49:20,944 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-04-02 08:49:20,944 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "JBOSS_HOME" is "/usr/share/jboss-as". >2014-04-02 08:49:20,945 INFO [org.ovirt.engine.core.utils.LocalConfig] (MSC service thread 1-13) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-04-02 08:49:21,460 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-04-02 08:49:21,885 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.4.0-1.el6 >2014-04-02 08:49:21,887 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-04-02 08:49:21,893 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-04-02 08:49:21,895 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-04-02 08:49:21,896 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-04-02 08:49:21,918 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-04-02 08:49:21,923 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-04-02 08:49:21,938 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-04-02 08:49:21,954 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-04-02 08:49:21,959 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-04-02 08:49:21,967 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-04-02 08:49:21,981 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-04-02 08:49:22,264 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-04-02 08:49:22,266 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-04-02 08:49:22,280 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-04-02 08:49:22,282 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-04-02 08:49:22,283 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-04-02 08:49:22,284 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-04-02 08:49:22,285 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-04-02 08:49:22,318 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-04-02 08:49:22,319 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-04-02 08:49:22,321 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-04-02 08:49:22,323 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-04-02 08:49:22,325 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-04-02 08:49:22,344 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-04-02 08:49:22,417 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-3) Start initializing UsersDomainsCacheManagerService >2014-04-02 08:49:22,433 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager >2014-04-02 08:49:22,435 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager >2014-04-02 08:49:22,436 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-3) Finished initializing UsersDomainsCacheManagerService >2014-04-02 08:49:22,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (MSC service thread 1-15) Initialization of AsyncTaskManager completed successfully. >2014-04-02 08:49:22,456 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) Start initializing ResourceManager >2014-04-02 08:49:22,701 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Entered VdsManager constructor >2014-04-02 08:49:22,735 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [39db3d08] Running command: LogoutUserCommand internal: false. >2014-04-02 08:49:22,736 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Initialize vdsBroker (192.168.71.22,54,321) >2014-04-02 08:49:22,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: UNASSIGNED not exist in string table >2014-04-02 08:49:22,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-04-02 08:49:22,802 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-04-02 08:49:22,804 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-04-02 08:49:22,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-04-02 08:49:22,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: DWH_STOPPED not exist in string table >2014-04-02 08:49:22,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: DWH_STARTED not exist in string table >2014-04-02 08:49:22,810 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: DWH_ERROR not exist in string table >2014-04-02 08:49:22,823 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_REMOVE_VM not have severity. Assumed Normal >2014-04-02 08:49:22,825 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2014-04-02 08:49:22,826 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-02 08:49:22,828 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-02 08:49:22,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2014-04-02 08:49:22,831 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI not have severity. Assumed Normal >2014-04-02 08:49:22,833 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: GLUSTER_SERVICES_LIST_NOT_FETCHED not have severity. Assumed Normal >2014-04-02 08:49:22,834 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: GLUSTER_VOLUME_BRICK_ADDED not have severity. Assumed Normal >2014-04-02 08:49:22,836 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_EXTEND_DISK_SIZE_UPDATE_VM_FAILURE not have severity. Assumed Normal >2014-04-02 08:49:22,837 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2014-04-02 08:49:22,839 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2014-04-02 08:49:22,841 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2014-04-02 08:49:22,842 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2014-04-02 08:49:22,844 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_ADDED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,845 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_TO_ADD_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,847 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_UPDATED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,848 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_TO_UPDATE_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,850 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_REMOVED_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,851 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] AuditLogType: USER_FAILED_TO_REMOVE_AFFINITY_GROUP not have severity. Assumed Normal >2014-04-02 08:49:22,864 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) VDS ff474b41-22c5-440e-8052-4cf40c27b250 was added to the Resource Manager >2014-04-02 08:49:22,865 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Entered VdsManager constructor >2014-04-02 08:49:22,878 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-15) Initialize vdsBroker (192.168.71.23,54,321) >2014-04-02 08:49:22,884 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) VDS 80910869-bf75-4485-9fe6-f1c73a1e667d was added to the Resource Manager >2014-04-02 08:49:22,915 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-15) Finished initializing ResourceManager >2014-04-02 08:49:22,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-15) Initialization of OvfDataUpdater completed successfully. >2014-04-02 08:49:22,928 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Start scheduling to enable vds load balancer >2014-04-02 08:49:22,930 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Finished scheduling to enable vds load balancer >2014-04-02 08:49:22,931 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Start HA Reservation check >2014-04-02 08:49:22,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [39db3d08] Correlation ID: 39db3d08, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-04-02 08:49:22,932 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Finished HA Reservation check >2014-04-02 08:49:22,944 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (org.ovirt.thread.pool-6-thread-1) MacPoolManager(70e7dfb3): Start initializing >2014-04-02 08:49:22,962 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-15) Init VM custom properties utilities >2014-04-02 08:49:22,964 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-15) Init device custom properties utilities >2014-04-02 08:49:22,969 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Initializing Scheduling manager >2014-04-02 08:49:22,973 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (org.ovirt.thread.pool-6-thread-1) MacPoolManager(70e7dfb3): Finished initializing. Available MACs in pool: 253 >2014-04-02 08:49:23,010 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) External scheduler disabled, discovery skipped >2014-04-02 08:49:23,011 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-15) Initialized Scheduling manager >2014-04-02 08:49:23,013 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-15) Initializing DWH Heart Beat >2014-04-02 08:49:23,015 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-15) DWH Heart Beat initialized >2014-04-02 08:49:26,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetStatsVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vds=Host[Node2]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-04-02 08:49:26,329 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) vds::refreshVdsStats Failed getVdsStats, vds = 80910869-bf75-4485-9fe6-f1c73a1e667d : Node2, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-04-02 08:49:26,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand.executeVdsBrokerCommand(GetStatsVDSCommand.java:25) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:519) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:480) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:337) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host Node2 is initializing. Message: Recovering from crash or Initializing >2014-04-02 08:49:26,639 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Failed to refresh VDS , vds = 80910869-bf75-4485-9fe6-f1c73a1e667d : Node2, error = Recovering from crash or Initializing, continuing. >2014-04-02 08:49:26,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]), log id: 25ca40b1 >2014-04-02 08:49:26,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 25ca40b1 >2014-04-02 08:49:26,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host Node1. >2014-04-02 08:49:27,016 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: Node1 >2014-04-02 08:49:27,054 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [315d6caa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:49:27,106 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-8) domain 43daa22b-28b8-4d67-9e43-77f22af1479e:ISOs in problem. vds: Node1 >2014-04-02 08:49:27,109 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-8) domain 220dcfb6-03c2-4520-b6a0-3e7dcae78af1:KVM-VMPOOL02 in problem. vds: Node1 >2014-04-02 08:49:27,112 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-8) domain b03ccc5e-9448-4c1d-959d-44028daa1c5d:KVM-VMPOOL01 in problem. vds: Node1 >2014-04-02 08:49:29,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GetHardwareInfoVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vds=Host[Node2]), log id: 7d4aed65 >2014-04-02 08:49:30,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GetHardwareInfoVDSCommand, log id: 7d4aed65 >2014-04-02 08:49:30,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GetHardwareInfoVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vds=Host[Node2]), log id: 4f74b657 >2014-04-02 08:49:30,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GetHardwareInfoVDSCommand, log id: 4f74b657 >2014-04-02 08:49:30,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host Node2. >2014-04-02 08:49:30,271 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Cancelling the recovery from crash timer for VDS 80910869-bf75-4485-9fe6-f1c73a1e667d because vds started initializing >2014-04-02 08:49:30,285 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [fe66130] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000000ea Type: StoragePool >2014-04-02 08:49:30,309 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000000ea Type: StoragePool >2014-04-02 08:49:30,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] START, ConnectStorageServerVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, storageType = NFS, connectionList = [{ id: 400c0dc0-4c94-43a5-a3ef-f8c25acedf1c, connection: 192.168.71.21:/home/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2b7aa9c1 >2014-04-02 08:49:30,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] FINISH, ConnectStorageServerVDSCommand, return: {400c0dc0-4c94-43a5-a3ef-f8c25acedf1c=0}, log id: 2b7aa9c1 >2014-04-02 08:49:30,739 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] Host Node2 storage connection was succeeded >2014-04-02 08:49:30,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) START, ConnectStoragePoolVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, vds_spm_id = 2, masterDomainId = b03ccc5e-9448-4c1d-959d-44028daa1c5d, masterVersion = 1), log id: 630ae774 >2014-04-02 08:49:32,926 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000000ea Type: StoragePool >2014-04-02 08:49:33,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [15d0ddb] Correlation ID: 15d0ddb, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-04-02 08:49:33,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] hostFromVds::selectedVds - Node1, spmStatus Unknown_Pool, storage pool Default >2014-04-02 08:49:33,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] START, ConnectStoragePoolVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea, vds_spm_id = 1, masterDomainId = b03ccc5e-9448-4c1d-959d-44028daa1c5d, masterVersion = 1), log id: 36268b9d >2014-04-02 08:49:34,470 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-04-02 08:49:34,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-04-02 08:49:35,368 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-04-02 08:49:35,463 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (ajp--127.0.0.1-8702-8) Can't read file "/var/lib/ovirt-engine/reports.xml" for request "/ovirt-engine/services/reports-ui", will send a 404 error response. >2014-04-02 08:49:50,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] FINISH, ConnectStoragePoolVDSCommand, log id: 36268b9d >2014-04-02 08:49:50,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] hostFromVds::Connected host to pool - selectedVds - Node1, spmStatus Free, storage pool Default >2014-04-02 08:49:50,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] starting spm on vds Node1, storage pool Default, prevId -1, LVER -1 >2014-04-02 08:49:51,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] START, SpmStartVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23fdef01 >2014-04-02 08:49:51,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] spmStart polling started: taskId = 7cd35525-67a2-40af-909d-7e0aaf8b5d07 >2014-04-02 08:49:56,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) FINISH, ConnectStoragePoolVDSCommand, log id: 630ae774 >2014-04-02 08:49:56,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain 43daa22b-28b8-4d67-9e43-77f22af1479e:ISOs recovered from problem. vds: Node1 >2014-04-02 08:49:56,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain 43daa22b-28b8-4d67-9e43-77f22af1479e:ISOs has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-04-02 08:49:56,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain 220dcfb6-03c2-4520-b6a0-3e7dcae78af1:KVM-VMPOOL02 recovered from problem. vds: Node1 >2014-04-02 08:49:56,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain 220dcfb6-03c2-4520-b6a0-3e7dcae78af1:KVM-VMPOOL02 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-04-02 08:49:56,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain b03ccc5e-9448-4c1d-959d-44028daa1c5d:KVM-VMPOOL01 recovered from problem. vds: Node1 >2014-04-02 08:49:56,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-36) Domain b03ccc5e-9448-4c1d-959d-44028daa1c5d:KVM-VMPOOL01 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-04-02 08:49:56,372 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [1252641c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node1 from cluster BCE was chosen as a proxy to execute Status command on Host Node2. >2014-04-02 08:49:56,373 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) [1252641c] Using Host Node1 from cluster BCE as proxy to execute Status command on Host Node2 >2014-04-02 08:49:56,400 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) [1252641c] Executing <Status> Power Management command, Proxy Host:Node1, Agent:bladecenter, Target Host:Node2, Management IP:192.168.71.18, User:USERID, Options:slot=3,secure=false >2014-04-02 08:49:56,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] START, FenceVdsVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, targetVdsId = 80910869-bf75-4485-9fe6-f1c73a1e667d, action = Status, ip = 192.168.71.18, port = , type = bladecenter, user = USERID, password = ******, options = 'slot=3,secure=false'), log id: 3ce322af >2014-04-02 08:49:57,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3ce322af >2014-04-02 08:49:57,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] START, SetMOMPolicyParametersVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d), log id: 56cadadf >2014-04-02 08:49:57,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [1252641c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 56cadadf >2014-04-02 08:49:58,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [1252641c] Correlation ID: fe66130, Call Stack: null, Custom Event ID: -1, Message: Host Node2 power management was verified successfully. >2014-04-02 08:49:58,040 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [1252641c] Initializing Host: Node2 >2014-04-02 08:49:58,069 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [f03319f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 80910869-bf75-4485-9fe6-f1c73a1e667d Type: VDS >2014-04-02 08:49:58,101 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [67ddb166] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 80910869-bf75-4485-9fe6-f1c73a1e667d Type: VDS >2014-04-02 08:50:18,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] spmStart polling ended: taskId = 7cd35525-67a2-40af-909d-7e0aaf8b5d07 task status = finished >2014-04-02 08:50:18,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] spmStart polling ended, spm status: SPM >2014-04-02 08:50:18,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] START, HSMClearTaskVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, taskId=7cd35525-67a2-40af-909d-7e0aaf8b5d07), log id: 64fc3917 >2014-04-02 08:50:18,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] FINISH, HSMClearTaskVDSCommand, log id: 64fc3917 >2014-04-02 08:50:18,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e50c4e3, log id: 23fdef01 >2014-04-02 08:50:18,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) [15d0ddb] Initialize Irs proxy from vds: 192.168.71.22 >2014-04-02 08:50:18,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [15d0ddb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host Node1 (Address: 192.168.71.22). >2014-04-02 08:50:18,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false), log id: 6162ff43 >2014-04-02 08:50:18,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000000ea >2014-04-02 08:50:18,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] START, HSMGetAllTasksInfoVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250), log id: 306c235b >2014-04-02 08:50:18,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 306c235b >2014-04-02 08:50:18,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6162ff43 >2014-04-02 08:50:18,870 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-33) [15d0ddb] Discovered no tasks on Storage Pool Default >2014-04-02 08:50:23,009 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-23) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:50:23,011 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-23) There is no over-utilized host in cluster BCE >2014-04-02 08:50:25,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain KVM-VMPOOL01 experienced a high latency of 8.21258 seconds from host Node2. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-04-02 08:50:51,796 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (ajp--127.0.0.1-8702-8) Can't read file "/var/lib/ovirt-engine/reports.xml" for request "/ovirt-engine/services/reports-ui", will send a 404 error response. >2014-04-02 08:51:23,086 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-65) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:51:23,088 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-65) There is no over-utilized host in cluster BCE >2014-04-02 08:52:23,140 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-2) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:52:23,142 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-2) There is no over-utilized host in cluster BCE >2014-04-02 08:53:23,195 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-21) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:53:23,197 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-21) There is no over-utilized host in cluster BCE >2014-04-02 08:54:21,976 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-39) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-04-02 08:54:21,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-41) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false), log id: 58bf00a6 >2014-04-02 08:54:21,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-39) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false), log id: 4865a08d >2014-04-02 08:54:23,247 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-42) There is no host with more than 10 running guests, no balancing is needed >2014-04-02 08:54:23,249 INFO [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-42) There is no over-utilized host in cluster BCE >2014-04-02 08:55:11,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-74) Command ListVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-02 08:55:11,109 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Failed to refresh VDS , vds = ff474b41-22c5-440e-8052-4cf40c27b250 : Node1, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-04-02 08:55:16,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:55:16,148 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Failed to refresh VDS , vds = ff474b41-22c5-440e-8052-4cf40c27b250 : Node1, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-04-02 08:55:17,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) Command SpmStatusVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-02 08:55:17,255 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000000ea Type: StoragePool >2014-04-02 08:55:17,325 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] Correlation ID: 5bc9ca14, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host Node1, Error: Network error during communication with the Host.). >2014-04-02 08:55:20,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:55:20,335 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Server failed to respond, vds_id = ff474b41-22c5-440e-8052-4cf40c27b250, vds_name = Node1, vm_count = 3, spm_status = SPM, non-responsive_timeout (seconds) = 81, error = java.net.NoRouteToHostException: No route to host >2014-04-02 08:55:20,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:102) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:537) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:490) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:337) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.NoRouteToHostException: No route to host > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy54.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:225) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host Node1 is non responsive. >2014-04-02 08:55:20,412 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-30) ResourceManager::vdsNotResponding entered for Host ff474b41-22c5-440e-8052-4cf40c27b250, 192.168.71.22 >2014-04-02 08:55:20,508 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-30) [3f93d420] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:55:20,546 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-30) [3f93d420] Opening SSH Soft Fencing session on host 192.168.71.22 >2014-04-02 08:55:23,332 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-30) [3f93d420] SSH Soft Fencing command failed on host 192.168.71.22: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.utils.ssh.SSHClient.connect(SSHClient.java:294) [utils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:77) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:44) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1123) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1208) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1884) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1228) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:413) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:392) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:634) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$1.run(VdsEventListener.java:144) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:97) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_51] > at org.apache.mina.transport.socket.nio.NioSocketConnector.finishConnect(NioSocketConnector.java:223) [mina-core.jar:] > at org.apache.mina.transport.socket.nio.NioSocketConnector.finishConnect(NioSocketConnector.java:45) [mina-core.jar:] > at org.apache.mina.core.polling.AbstractPollingIoConnector.processConnections(AbstractPollingIoConnector.java:427) [mina-core.jar:] > at org.apache.mina.core.polling.AbstractPollingIoConnector.access$500(AbstractPollingIoConnector.java:62) [mina-core.jar:] > at org.apache.mina.core.polling.AbstractPollingIoConnector$Connector.run(AbstractPollingIoConnector.java:476) [mina-core.jar:] > at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [mina-core.jar:] > ... 3 more > >2014-04-02 08:55:23,458 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-30) [517d8b43] Lock Acquired to object EngineLock [exclusiveLocks= key: ff474b41-22c5-440e-8052-4cf40c27b250 value: VDS_FENCE >, sharedLocks= ] >2014-04-02 08:55:23,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Restart command on Host Node1. >2014-04-02 08:55:23,577 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Using Host Node2 from cluster BCE as proxy to execute Restart command on Host Node1 >2014-04-02 08:55:23,600 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-30) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:55:23,692 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Running command: StopVdsCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:55:23,720 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SetVdsStatusVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 50cce261 >2014-04-02 08:55:25,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:55:25,493 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] VDS Node1 is spm and moved from up calling resetIrs. >2014-04-02 08:55:25,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false, vdsId = ff474b41-22c5-440e-8052-4cf40c27b250, ignoreStopFailed = false), log id: 228e0b4c >2014-04-02 08:55:25,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SpmStopVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea), log id: 287e53f3 >2014-04-02 08:55:25,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] SpmStopVDSCommand:: vds Node1 is in NonResponsive status - not performing spm stop, pool id 00000002-0002-0002-0002-0000000000ea >2014-04-02 08:55:25,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SpmStopVDSCommand, log id: 287e53f3 >2014-04-02 08:55:25,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, ResetIrsVDSCommand, log id: 228e0b4c >2014-04-02 08:55:25,542 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SetVdsStatusVDSCommand, log id: 50cce261 >2014-04-02 08:55:25,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Stop command on Host Node1. >2014-04-02 08:55:25,694 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Using Host Node2 from cluster BCE as proxy to execute Stop command on Host Node1 >2014-04-02 08:55:25,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SpmStopVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea), log id: 7592b13a >2014-04-02 08:55:25,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] SpmStopVDSCommand:: vds Node1 is in Reboot status - not performing spm stop, pool id 00000002-0002-0002-0002-0000000000ea >2014-04-02 08:55:25,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SpmStopVDSCommand, log id: 7592b13a >2014-04-02 08:55:25,748 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Executing <Stop> Power Management command, Proxy Host:Node2, Agent:bladecenter, Target Host:Node1, Management IP:192.168.71.18, User:USERID, Options:slot=1,secure=false >2014-04-02 08:55:25,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, FenceVdsVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, targetVdsId = ff474b41-22c5-440e-8052-4cf40c27b250, action = Stop, ip = 192.168.71.18, port = , type = bladecenter, user = USERID, password = ******, options = 'slot=1,secure=false'), log id: 71b19323 >2014-04-02 08:55:26,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node1 is already off, Power Management Stop operation skipped. >2014-04-02 08:55:26,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, FenceVdsVDSCommand, return: Test Succeeded, skipped, log id: 71b19323 >2014-04-02 08:55:26,545 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Waiting for vds Node1 to stop >2014-04-02 08:55:31,547 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Attempt 1 to get vds Node1 status >2014-04-02 08:55:31,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Status command on Host Node1. >2014-04-02 08:55:31,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Using Host Node2 from cluster BCE as proxy to execute Status command on Host Node1 >2014-04-02 08:55:31,803 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Executing <Status> Power Management command, Proxy Host:Node2, Agent:bladecenter, Target Host:Node1, Management IP:192.168.71.18, User:USERID, Options:slot=1,secure=false >2014-04-02 08:55:31,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, FenceVdsVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, targetVdsId = ff474b41-22c5-440e-8052-4cf40c27b250, action = Status, ip = 192.168.71.18, port = , type = bladecenter, user = USERID, password = ******, options = 'slot=1,secure=false'), log id: 21d0423a >2014-04-02 08:55:33,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 21d0423a >2014-04-02 08:55:33,188 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] vds Node1 status is off >2014-04-02 08:55:33,190 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SetVmStatusVDSCommand( vmId = 21d83332-ef88-4c6d-b59b-b207aee7a983, status = Down), log id: 155baf04 >2014-04-02 08:55:33,224 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SetVmStatusVDSCommand, log id: 155baf04 >2014-04-02 08:55:33,316 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm WIN701 was shut down due to Node1 host reboot or manual fence >2014-04-02 08:55:33,368 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SetVmStatusVDSCommand( vmId = 69a95dd5-16e3-437e-b1e8-3c6e68a344f8, status = Down), log id: 13538eb4 >2014-04-02 08:55:33,391 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SetVmStatusVDSCommand, log id: 13538eb4 >2014-04-02 08:55:33,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm W2K08R2-01 was shut down due to Node1 host reboot or manual fence >2014-04-02 08:55:33,442 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, SetVmStatusVDSCommand( vmId = 5e859cb9-ec08-45b1-8e51-fb7e66c72011, status = Down), log id: 5abeb36c >2014-04-02 08:55:33,458 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, SetVmStatusVDSCommand, log id: 5abeb36c >2014-04-02 08:55:33,500 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm WIN702 was shut down due to Node1 host reboot or manual fence >2014-04-02 08:55:33,582 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-04-02 08:55:33,583 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [4e48cdbb] Start running CanDoAction for command number 3/3 (Command type: RunVm) >2014-04-02 08:55:33,582 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-04-02 08:55:33,585 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [4e48cdbb] Start running CanDoAction for command number 1/3 (Command type: RunVm) >2014-04-02 08:55:33,587 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-44) [4e48cdbb] Start running CanDoAction for command number 2/3 (Command type: RunVm) >2014-04-02 08:55:33,593 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-45) [4e48cdbb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e859cb9-ec08-45b1-8e51-fb7e66c72011 value: VM >, sharedLocks= ] >2014-04-02 08:55:33,593 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-42) [4e48cdbb] Lock Acquired to object EngineLock [exclusiveLocks= key: 21d83332-ef88-4c6d-b59b-b207aee7a983 value: VM >, sharedLocks= ] >2014-04-02 08:55:33,594 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-44) [4e48cdbb] Lock Acquired to object EngineLock [exclusiveLocks= key: 69a95dd5-16e3-437e-b1e8-3c6e68a344f8 value: VM >, sharedLocks= ] >2014-04-02 08:55:33,680 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-45) [4e48cdbb] START, IsVmDuringInitiatingVDSCommand( vmId = 5e859cb9-ec08-45b1-8e51-fb7e66c72011), log id: 1a9ed855 >2014-04-02 08:55:33,680 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-42) [4e48cdbb] START, IsVmDuringInitiatingVDSCommand( vmId = 21d83332-ef88-4c6d-b59b-b207aee7a983), log id: 328cc21a >2014-04-02 08:55:33,683 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-45) [4e48cdbb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1a9ed855 >2014-04-02 08:55:33,684 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-42) [4e48cdbb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 328cc21a >2014-04-02 08:55:33,695 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-44) [4e48cdbb] START, IsVmDuringInitiatingVDSCommand( vmId = 69a95dd5-16e3-437e-b1e8-3c6e68a344f8), log id: 19142ee7 >2014-04-02 08:55:33,697 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-44) [4e48cdbb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 19142ee7 >2014-04-02 08:55:33,747 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [4e48cdbb] Finish handling CanDoAction for command number 3/3 (Command type: RunVm) >2014-04-02 08:55:33,747 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [4e48cdbb] Finish handling CanDoAction for command number 1/3 (Command type: RunVm) >2014-04-02 08:55:33,766 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-44) [4e48cdbb] Finish handling CanDoAction for command number 2/3 (Command type: RunVm) >2014-04-02 08:55:33,774 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Running command: RunVmCommand internal: true. Entities affected : ID: 69a95dd5-16e3-437e-b1e8-3c6e68a344f8 Type: VM >2014-04-02 08:55:33,794 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] START, UpdateVdsVMsClearedVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250), log id: 1ea3f01b >2014-04-02 08:55:33,800 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 1ea3f01b >2014-04-02 08:55:33,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4e48cdbb] Correlation ID: 4e48cdbb, Job ID: 53f79a4e-efa7-4aaf-a503-c70531e84cf0, Call Stack: null, Custom Event ID: -1, Message: Host Node1 was stopped by Engine. >2014-04-02 08:55:33,918 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Running VM with attached cd SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008_R2_64Bit_German_w_SP1_MLF_X17-22588.ISO >2014-04-02 08:55:33,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, IsoPrefixVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId=00000002-0002-0002-0002-0000000000ea), log id: 5b952f08 >2014-04-02 08:55:33,950 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:55:34,109 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Trying to fence spm Node1 via vds Node2 >2014-04-02 08:55:34,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] START, FenceSpmStorageVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, prevId=1, prevLVER=74), log id: 24f60dce >2014-04-02 08:55:34,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/192.168.71.21:_home_iso/43daa22b-28b8-4d67-9e43-77f22af1479e/images/11111111-1111-1111-1111-111111111111, log id: 5b952f08 >2014-04-02 08:55:34,231 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVmVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, vm=VM [W2K08R2-01]), log id: a9a51ea >2014-04-02 08:55:34,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Failed in FenceSpmStorageVDS method >2014-04-02 08:55:34,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=655, mMessage=Error fencing SPM: ('spUUID=00000002-0002-0002-0002-0000000000ea, lastOwner=None, lastLver=None',)]] >2014-04-02 08:55:34,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] HostName = Node2 >2014-04-02 08:55:34,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Command FenceSpmStorageVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, prevId=1, prevLVER=74) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to FenceSpmStorageVDS, error = Error fencing SPM: ('spUUID=00000002-0002-0002-0002-0000000000ea, lastOwner=None, lastLver=None',), code = 655 >2014-04-02 08:55:34,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] FINISH, FenceSpmStorageVDSCommand, log id: 24f60dce >2014-04-02 08:55:34,247 WARN [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Could not fence spm on vds Node2 >2014-04-02 08:55:34,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, vm=VM [W2K08R2-01]), log id: 39fa81d2 >2014-04-02 08:55:34,347 ERROR [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand. >2014-04-02 08:55:34,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [114dd04c] Correlation ID: 114dd04c, Job ID: 10d34c29-e95c-4ecf-a639-1766efcaafb7, Call Stack: null, Custom Event ID: -1, Message: Manual fence did not revoke the selected SPM (Node1) since the master storage domain > was not active or could not use another host for the fence operation. >2014-04-02 08:55:34,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=8000,kvmEnable=true,smp=8,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=de,memGuaranteedSize=5333,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=4,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=f7e9318c-acbe-4dac-bc11-3daecf280459}, {shared=false, bootOrder=1, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008_R2_64Bit_German_w_SP1_MLF_X17-22588.ISO}, path=/rhev/data-center/mnt/192.168.71.21:_home_iso/43daa22b-28b8-4d67-9e43-77f22af1479e/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008_R2_64Bit_German_w_SP1_MLF_X17-22588.ISO, device=cdrom, type=disk, readonly=true, deviceId=09a4de08-1439-4445-a6c1-335d4def4def}, {shared=false, index=0, volumeID=65ab3b75-070b-4e27-8cb2-ce3ee36b4222, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=2, address={bus=0x00, domain=0x0000, slot=0x05, type=pci, function=0x0}, domainID=b03ccc5e-9448-4c1d-959d-44028daa1c5d, imageID=314dcae0-fbcd-443e-ba5f-5110a920e4b3, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000000ea, readonly=false, deviceId=314dcae0-fbcd-443e-ba5f-5110a920e4b3}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, macAddr=00:1a:4a:ec:55:95, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=ed251e4d-f82d-4508-a283-2a2c8977dec0}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=50573d39-ff99-44e8-a9b7-e83672dad255}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=81a9b6f4-eb28-4b43-ae4f-b3bb593e0190}],acpiEnable=true,vmName=W2K08R2-01,cpuType=Penryn,custom={device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fc=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=1eb710b2-3c1c-4cd8-9316-4ee6557264fc, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22device_ddab3939-a463-4c59-8250-12510165ebeadevice_6bd5531a-7855-4085-8f6d-af4cd1e1c501=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=6bd5531a-7855-4085-8f6d-af4cd1e1c501, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22device_ddab3939-a463-4c59-8250-12510165ebea=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=ddab3939-a463-4c59-8250-12510165ebea, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=9abf8224-d72e-43fa-96c0-3d5f04388d22, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97f=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=c25c8f15-31aa-480d-a21e-d4535b79c97f, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}} >2014-04-02 08:55:34,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVDSCommand, log id: 39fa81d2 >2014-04-02 08:55:34,467 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: a9a51ea >2014-04-02 08:55:34,468 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Lock freed to object EngineLock [exclusiveLocks= key: 69a95dd5-16e3-437e-b1e8-3c6e68a344f8 value: VM >, sharedLocks= ] >2014-04-02 08:55:34,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM W2K08R2-01 was restarted on Host Node2 >2014-04-02 08:55:34,494 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Running command: RunVmCommand internal: true. Entities affected : ID: 5e859cb9-ec08-45b1-8e51-fb7e66c72011 Type: VM >2014-04-02 08:55:34,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Status command on Host Node1. >2014-04-02 08:55:34,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Using Host Node2 from cluster BCE as proxy to execute Status command on Host Node1 >2014-04-02 08:55:34,600 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-30) Running command: StartVdsCommand internal: true. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:55:34,627 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) START, SetVdsStatusVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5ac06f01 >2014-04-02 08:55:34,628 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) VDS Node1 is spm and moved from up calling resetIrs. >2014-04-02 08:55:34,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-30) START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ea, ignoreFailoverLimit = false, vdsId = ff474b41-22c5-440e-8052-4cf40c27b250, ignoreStopFailed = false), log id: 481ad133 >2014-04-02 08:55:34,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) START, SpmStopVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea), log id: 2e55d9ef >2014-04-02 08:55:34,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) SpmStopVDSCommand:: vds Node1 is in Reboot status - not performing spm stop, pool id 00000002-0002-0002-0002-0000000000ea >2014-04-02 08:55:34,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-30) FINISH, SpmStopVDSCommand, log id: 2e55d9ef >2014-04-02 08:55:34,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-30) FINISH, ResetIrsVDSCommand, log id: 481ad133 >2014-04-02 08:55:34,667 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) FINISH, SetVdsStatusVDSCommand, log id: 5ac06f01 >2014-04-02 08:55:34,688 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVmVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, vm=VM [WIN702]), log id: 41edde77 >2014-04-02 08:55:34,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, vm=VM [WIN702]), log id: 69c9b2f6 >2014-04-02 08:55:34,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Start command on Host Node1. >2014-04-02 08:55:34,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Using Host Node2 from cluster BCE as proxy to execute Start command on Host Node1 >2014-04-02 08:55:34,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=8000,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=de,memGuaranteedSize=5333,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=4,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=ba60bb38-0a17-4d63-aea0-1b992edc5b2b}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=9d3cb9b3-e3c8-42ac-b210-7e49aeb22d07}, {shared=false, index=0, volumeID=75212200-38b2-44f1-bd4b-244f408bdae9, propagateErrors=off, format=cow, type=disk, iface=virtio, bootOrder=1, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=b03ccc5e-9448-4c1d-959d-44028daa1c5d, imageID=da9afb49-b44f-44f2-9f97-828c36cd218e, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000000ea, readonly=false, deviceId=da9afb49-b44f-44f2-9f97-828c36cd218e}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x08, type=pci, function=0x0}, specParams={}, macAddr=00:1a:4a:ec:55:90, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=89b7b6e4-6be7-42fe-a95f-8467760ddb2b}, {address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={}, device=ich6, type=sound, deviceId=68ecd551-3802-4121-8cb0-92abffe86886}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=420323fd-68ac-4bca-b9ff-2316a49c3cdb}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=f42855f1-ae0a-422c-8243-8ba8552877f7}],acpiEnable=true,vmName=WIN702,cpuType=Penryn,custom={device_b9dcbd68-8976-44f1-84ee-ad588472a36adevice_abbc82ce-0faa-468c-84e0-6572ef04ff82device_c5689f98-3e24-4480-96b5-b3b21cf4247b=VmDevice {vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, deviceId=c5689f98-3e24-4480-96b5-b3b21cf4247b, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_b9dcbd68-8976-44f1-84ee-ad588472a36adevice_abbc82ce-0faa-468c-84e0-6572ef04ff82device_c5689f98-3e24-4480-96b5-b3b21cf4247bdevice_0ba909b1-1e94-4c22-952d-04c1081b4665device_e3abcc78-b680-453f-a72a-b5b1d238c613=VmDevice {vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, deviceId=e3abcc78-b680-453f-a72a-b5b1d238c613, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_b9dcbd68-8976-44f1-84ee-ad588472a36a=VmDevice {vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, deviceId=b9dcbd68-8976-44f1-84ee-ad588472a36a, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_b9dcbd68-8976-44f1-84ee-ad588472a36adevice_abbc82ce-0faa-468c-84e0-6572ef04ff82=VmDevice {vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, deviceId=abbc82ce-0faa-468c-84e0-6572ef04ff82, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_b9dcbd68-8976-44f1-84ee-ad588472a36adevice_abbc82ce-0faa-468c-84e0-6572ef04ff82device_c5689f98-3e24-4480-96b5-b3b21cf4247bdevice_0ba909b1-1e94-4c22-952d-04c1081b4665=VmDevice {vmId=5e859cb9-ec08-45b1-8e51-fb7e66c72011, deviceId=0ba909b1-1e94-4c22-952d-04c1081b4665, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}} >2014-04-02 08:55:34,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVDSCommand, log id: 69c9b2f6 >2014-04-02 08:55:34,806 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Executing <Start> Power Management command, Proxy Host:Node2, Agent:bladecenter, Target Host:Node1, Management IP:192.168.71.18, User:USERID, Options:slot=1,secure=false >2014-04-02 08:55:34,817 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 41edde77 >2014-04-02 08:55:34,818 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Lock freed to object EngineLock [exclusiveLocks= key: 5e859cb9-ec08-45b1-8e51-fb7e66c72011 value: VM >, sharedLocks= ] >2014-04-02 08:55:34,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) START, FenceVdsVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, targetVdsId = ff474b41-22c5-440e-8052-4cf40c27b250, action = Start, ip = 192.168.71.18, port = , type = bladecenter, user = USERID, password = ******, options = 'slot=1,secure=false'), log id: 3b0ede9b >2014-04-02 08:55:34,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM WIN702 was restarted on Host Node2 >2014-04-02 08:55:34,836 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Running command: RunVmCommand internal: true. Entities affected : ID: 21d83332-ef88-4c6d-b59b-b207aee7a983 Type: VM >2014-04-02 08:55:34,984 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVmVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, vm=VM [WIN701]), log id: 71be3f79 >2014-04-02 08:55:35,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] START, CreateVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, vm=VM [WIN701]), log id: 20b99fc >2014-04-02 08:55:35,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=8000,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=de,memGuaranteedSize=5333,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=4,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=21d83332-ef88-4c6d-b59b-b207aee7a983,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d13dcf02-0eb7-4d98-81ba-d1dfbd9e3017}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=fbf85f35-644f-4512-9ae1-f5bc46fd566f}, {shared=false, index=0, volumeID=2aef087f-ff2c-4acb-9984-2382a060081b, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=b03ccc5e-9448-4c1d-959d-44028daa1c5d, imageID=81205625-8af6-4d57-8f81-cab2432876b4, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000000ea, readonly=false, deviceId=81205625-8af6-4d57-8f81-cab2432876b4}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x08, type=pci, function=0x0}, specParams={}, macAddr=00:1a:4a:ec:55:91, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=7d1ed35a-8cb3-4e54-99ef-071812dc306a}, {address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={}, device=ich6, type=sound, deviceId=902e41de-9f91-4343-a868-cb9bff554a08}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=8d363334-c219-444a-9fc6-c25366bcad98}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=ec5dea64-b972-4362-b8fb-255539983043}],acpiEnable=true,vmName=WIN701,cpuType=Penryn,custom={device_8e814020-5697-43e8-bf04-74f7ca154ec4=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=8e814020-5697-43e8-bf04-74f7ca154ec4, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=06d0ae4d-ad55-4423-8ea4-9ab619949584, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584device_247e3bda-b264-44ff-b83e-57fd6945311adevice_56db96b4-87ae-4929-a1c9-a5179a6c2395=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=56db96b4-87ae-4929-a1c9-a5179a6c2395, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753b=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=a052cca2-0118-4b84-b58a-6365e066753b, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584device_247e3bda-b264-44ff-b83e-57fd6945311a=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=247e3bda-b264-44ff-b83e-57fd6945311a, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}} >2014-04-02 08:55:35,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVDSCommand, log id: 20b99fc >2014-04-02 08:55:35,117 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 71be3f79 >2014-04-02 08:55:35,118 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Lock freed to object EngineLock [exclusiveLocks= key: 21d83332-ef88-4c6d-b59b-b207aee7a983 value: VM >, sharedLocks= ] >2014-04-02 08:55:35,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [4e48cdbb] Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM WIN701 was restarted on Host Node2 >2014-04-02 08:55:35,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) FINISH, FenceVdsVDSCommand, return: Done, log id: 3b0ede9b >2014-04-02 08:55:35,641 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-30) Waiting for vds Node1 to start >2014-04-02 08:55:38,337 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:55:40,642 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-30) Attempt 1 to get vds Node1 status >2014-04-02 08:55:40,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host Node2 from cluster BCE was chosen as a proxy to execute Status command on Host Node1. >2014-04-02 08:55:40,752 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Using Host Node2 from cluster BCE as proxy to execute Status command on Host Node1 >2014-04-02 08:55:40,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-6-thread-30) Executing <Status> Power Management command, Proxy Host:Node2, Agent:bladecenter, Target Host:Node1, Management IP:192.168.71.18, User:USERID, Options:slot=1,secure=false >2014-04-02 08:55:40,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) START, FenceVdsVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, targetVdsId = ff474b41-22c5-440e-8052-4cf40c27b250, action = Status, ip = 192.168.71.18, port = , type = bladecenter, user = USERID, password = ******, options = 'slot=1,secure=false'), log id: 50b82810 >2014-04-02 08:55:41,050 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) VM W2K08R2-01 69a95dd5-16e3-437e-b1e8-3c6e68a344f8 moved from WaitForLaunch --> PoweringUp >2014-04-02 08:55:41,052 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) VM WIN702 5e859cb9-ec08-45b1-8e51-fb7e66c72011 moved from WaitForLaunch --> PoweringUp >2014-04-02 08:55:41,053 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) VM WIN701 21d83332-ef88-4c6d-b59b-b207aee7a983 moved from WaitForLaunch --> PoweringUp >2014-04-02 08:55:41,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-2) START, FullListVdsCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, vds=Host[Node2], vmIds=[69a95dd5-16e3-437e-b1e8-3c6e68a344f8, 21d83332-ef88-4c6d-b59b-b207aee7a983]), log id: 245e3269 >2014-04-02 08:55:41,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-2) FINISH, FullListVdsCommand, return: [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=de, displayIp=0, pauseCode=NOERR, nice=0, timeOffset=3600, transparentHugePages=true, vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, acpiEnable=true, custom={device_8e814020-5697-43e8-bf04-74f7ca154ec4=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=8e814020-5697-43e8-bf04-74f7ca154ec4, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=06d0ae4d-ad55-4423-8ea4-9ab619949584, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584device_247e3bda-b264-44ff-b83e-57fd6945311adevice_56db96b4-87ae-4929-a1c9-a5179a6c2395=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=56db96b4-87ae-4929-a1c9-a5179a6c2395, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753b=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=a052cca2-0118-4b84-b58a-6365e066753b, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_8e814020-5697-43e8-bf04-74f7ca154ec4device_a052cca2-0118-4b84-b58a-6365e066753bdevice_06d0ae4d-ad55-4423-8ea4-9ab619949584device_247e3bda-b264-44ff-b83e-57fd6945311a=VmDevice {vmId=21d83332-ef88-4c6d-b59b-b207aee7a983, deviceId=247e3bda-b264-44ff-b83e-57fd6945311a, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=8000, displayPort=5904, smp=4, emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, memGuaranteedSize=5333, display=qxl, pid=6300, smartcardEnable=false, smpCoresPerSocket=4, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@243f9e26, vmName=WIN701, cpuType=Penryn}, {displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=de, displayIp=0, pauseCode=NOERR, nice=0, timeOffset=3600, transparentHugePages=true, vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, acpiEnable=true, custom={device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fc=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=1eb710b2-3c1c-4cd8-9316-4ee6557264fc, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22device_ddab3939-a463-4c59-8250-12510165ebeadevice_6bd5531a-7855-4085-8f6d-af4cd1e1c501=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=6bd5531a-7855-4085-8f6d-af4cd1e1c501, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22device_ddab3939-a463-4c59-8250-12510165ebea=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=ddab3939-a463-4c59-8250-12510165ebea, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97fdevice_1eb710b2-3c1c-4cd8-9316-4ee6557264fcdevice_9abf8224-d72e-43fa-96c0-3d5f04388d22=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=9abf8224-d72e-43fa-96c0-3d5f04388d22, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_c25c8f15-31aa-480d-a21e-d4535b79c97f=VmDevice {vmId=69a95dd5-16e3-437e-b1e8-3c6e68a344f8, deviceId=c25c8f15-31aa-480d-a21e-d4535b79c97f, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=8000, displayPort=5900, smp=8, emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, memGuaranteedSize=5333, display=qxl, pid=6015, smartcardEnable=false, smpCoresPerSocket=4, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@4ca7e361, vmName=W2K08R2-01, cpuType=Penryn}], log id: 245e3269 >2014-04-02 08:55:42,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-30) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 50b82810 >2014-04-02 08:55:42,184 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-6-thread-30) vds Node1 status is on >2014-04-02 08:55:42,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: 621069d5, Job ID: 7686c502-417b-434a-8bb7-ec41fe026889, Call Stack: null, Custom Event ID: -1, Message: Host Node1 was started by Engine. >2014-04-02 08:55:42,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) Correlation ID: 517d8b43, Job ID: a65b5e4d-231d-4898-95c3-79ce4fc213be, Call Stack: null, Custom Event ID: -1, Message: Host Node1 is rebooting. >2014-04-02 08:55:42,242 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-30) Lock freed to object EngineLock [exclusiveLocks= key: ff474b41-22c5-440e-8052-4cf40c27b250 value: VDS_FENCE >, sharedLocks= ] >2014-04-02 08:55:43,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:55:47,339 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:55:52,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:55:56,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:01,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:05,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:10,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:14,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:19,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:23,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:28,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:32,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:33,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Used CPU of host Node2 [97%] exceeded defined threshold [95%]. >2014-04-02 08:56:36,878 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) VM W2K08R2-01 69a95dd5-16e3-437e-b1e8-3c6e68a344f8 moved from PoweringUp --> Up >2014-04-02 08:56:36,880 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) VM WIN702 5e859cb9-ec08-45b1-8e51-fb7e66c72011 moved from PoweringUp --> Up >2014-04-02 08:56:36,882 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) VM WIN701 21d83332-ef88-4c6d-b59b-b207aee7a983 moved from PoweringUp --> Up >2014-04-02 08:56:36,931 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM W2K08R2-01 was restarted on Host Node2 >2014-04-02 08:56:36,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM WIN702 was restarted on Host Node2 >2014-04-02 08:56:36,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: 4e48cdbb, Call Stack: null, Custom Event ID: -1, Message: VM WIN701 was restarted on Host Node2 >2014-04-02 08:56:37,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:41,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:46,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:50,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:56:55,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [67ddb166] Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:56:59,363 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:04,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:08,366 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:13,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:17,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:22,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:26,372 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:31,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:34,374 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-2) [5997178] Lock Acquired to object EngineLock [exclusiveLocks= key: ff474b41-22c5-440e-8052-4cf40c27b250 value: VDS_FENCE >, sharedLocks= ] >2014-04-02 08:57:34,402 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: ff474b41-22c5-440e-8052-4cf40c27b250 Type: VDS >2014-04-02 08:57:34,479 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Trying to fence spm Node1 via vds Node2 >2014-04-02 08:57:34,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] START, FenceSpmStorageVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, prevId=1, prevLVER=74), log id: 32474395 >2014-04-02 08:57:34,567 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Failed in FenceSpmStorageVDS method >2014-04-02 08:57:34,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Command org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=655, mMessage=Error fencing SPM: ('spUUID=00000002-0002-0002-0002-0000000000ea, lastOwner=None, lastLver=None',)]] >2014-04-02 08:57:34,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] HostName = Node2 >2014-04-02 08:57:34,571 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Command FenceSpmStorageVDSCommand(HostName = Node2, HostId = 80910869-bf75-4485-9fe6-f1c73a1e667d, storagePoolId = 00000002-0002-0002-0002-0000000000ea, prevId=1, prevLVER=74) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to FenceSpmStorageVDS, error = Error fencing SPM: ('spUUID=00000002-0002-0002-0002-0000000000ea, lastOwner=None, lastLver=None',), code = 655 >2014-04-02 08:57:34,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceSpmStorageVDSCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] FINISH, FenceSpmStorageVDSCommand, log id: 32474395 >2014-04-02 08:57:34,576 WARN [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Could not fence spm on vds Node2 >2014-04-02 08:57:34,669 ERROR [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand. >2014-04-02 08:57:34,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [5997178] Correlation ID: 5997178, Call Stack: null, Custom Event ID: -1, Message: Manual fence did not revoke the selected SPM (Node1) since the master storage domain > was not active or could not use another host for the fence operation. >2014-04-02 08:57:34,682 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-46) [5997178] Lock freed to object EngineLock [exclusiveLocks= key: ff474b41-22c5-440e-8052-4cf40c27b250 value: VDS_FENCE >, sharedLocks= ] >2014-04-02 08:57:35,375 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:40,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:44,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:49,452 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:57:53,381 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:57:58,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:58:02,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:58:07,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:58:11,389 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:58:16,444 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:58:17,329 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] IrsBroker::Failed::GetStoragePoolInfoVDS due to: TimeoutException: >2014-04-02 08:58:19,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] Command SpmStatusVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, storagePoolId = 00000002-0002-0002-0002-0000000000ea) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:58:19,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] Irs placed on server ff474b41-22c5-440e-8052-4cf40c27b250 failed. Proceed Failover >2014-04-02 08:58:19,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] hostFromVds::selectedVds - Node2, spmStatus Free, storage pool Default >2014-04-02 08:58:19,487 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] SPM Init: could not find reported vds or not up - pool:Default vds_spm_id: 1 >2014-04-02 08:58:19,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] SPM selection - vds seems as spm Node1 >2014-04-02 08:58:19,513 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [5bc9ca14] spm vds is non responsive, stopping spm selection. >2014-04-02 08:58:19,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) hostFromVds::selectedVds - Node2, spmStatus Free, storage pool Default >2014-04-02 08:58:19,558 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) SPM Init: could not find reported vds or not up - pool:Default vds_spm_id: 1 >2014-04-02 08:58:19,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) SPM selection - vds seems as spm Node1 >2014-04-02 08:58:19,584 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) spm vds is non responsive, stopping spm selection. >2014-04-02 08:58:19,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-39) FINISH, GetFileStatsVDSCommand, log id: 4865a08d >2014-04-02 08:58:19,585 WARN [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-39) The refresh process from VDSM, for Iso files failed. >2014-04-02 08:58:19,586 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-39) org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) >2014-04-02 08:58:19,588 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-39) Finished automatic refresh process for ISO file type with failure, for storage domain id 43daa22b-28b8-4d67-9e43-77f22af1479e. >2014-04-02 08:58:19,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-41) hostFromVds::selectedVds - Node2, spmStatus Free, storage pool Default >2014-04-02 08:58:19,638 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-41) SPM Init: could not find reported vds or not up - pool:Default vds_spm_id: 1 >2014-04-02 08:58:19,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-41) SPM selection - vds seems as spm Node1 >2014-04-02 08:58:19,664 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-41) spm vds is non responsive, stopping spm selection. >2014-04-02 08:58:19,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-41) FINISH, GetFileStatsVDSCommand, log id: 58bf00a6 >2014-04-02 08:58:19,666 WARN [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-41) The refresh process from VDSM, for Floppy files failed. >2014-04-02 08:58:19,667 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-41) org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) >2014-04-02 08:58:19,668 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-6-thread-41) Finished automatic refresh process for Floppy file type with failure, for storage domain id 43daa22b-28b8-4d67-9e43-77f22af1479e. >2014-04-02 08:58:19,744 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (DefaultQuartzScheduler_Worker-33) The following storage domains had a problem retrieving data from VDSM ( Storage Pool Id: 00000002-0002-0002-0002-0000000000ea Storage domain Id: 43daa22b-28b8-4d67-9e43-77f22af1479e File type: ISO) ( Storage Pool Id: 00000002-0002-0002-0002-0000000000ea Storage domain Id: 43daa22b-28b8-4d67-9e43-77f22af1479e File type: Floppy) >2014-04-02 08:58:19,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list failed for domain(s): ISOs (Floppy file type) ISOs (ISO file type). Please check domain activity. >2014-04-02 08:58:20,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-02 08:58:25,452 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-02 08:58:29,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) hostFromVds::selectedVds - Node2, spmStatus Free, storage pool Default >2014-04-02 08:58:29,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) SPM Init: could not find reported vds or not up - pool:Default vds_spm_id: 1 >2014-04-02 08:58:29,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) SPM selection - vds seems as spm Node1 >2014-04-02 08:58:29,585 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) spm vds is non responsive, stopping spm selection. >2014-04-02 08:58:30,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = Node1, HostId = ff474b41-22c5-440e-8052-4cf40c27b250, vds=Host[Node1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1082365
: 881647 |
881648
|
881649
|
881650
|
881651