Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 942291 Details for
Bug 1147487
[Fencing] stop vdsmd on one host is followed by engine connection problem to the host and to a second host, fencing flow isn't invoked
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
engine_29_09.log (text/plain), 1.19 MB, created by
sefi litmanovich
on 2014-09-29 11:32:22 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
sefi litmanovich
Created:
2014-09-29 11:32:22 UTC
Size:
1.19 MB
patch
obsolete
>2014-09-18 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-52) Start deleteAgedOutAuditLogs >2014-09-18 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) Start deleteAgedOutCommandEntities >2014-09-18 03:35:35,022 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-52) Finished deleteAgedOutAuditLogs >2014-09-18 03:35:35,025 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) Finished deleteAgedOutCommandEntities >2014-09-18 03:42:43,231 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-18 03:42:45,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-09-18 04:42:43,268 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-88) Principal admin::internal synchronized >2014-09-18 04:42:45,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [164567a] Attempting to update VMs/Templates Ovf. >2014-09-18 05:42:43,285 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-09-18 05:42:45,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-09-18 06:42:43,303 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-36) Principal admin::internal synchronized >2014-09-18 06:42:45,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-09-18 07:42:43,358 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) [76aaf268] Principal admin::internal synchronized >2014-09-18 07:42:45,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-09-18 08:42:43,370 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-09-18 08:42:45,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-09-18 09:42:43,412 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-09-18 09:42:45,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-09-18 10:18:53,270 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-12) Invalid locale found in configuration: >2014-09-18 10:18:53,270 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-12) Invalid locale found in configuration: >2014-09-18 10:42:43,421 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-09-18 10:42:45,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-09-18 11:42:43,434 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized >2014-09-18 11:42:45,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-09-18 12:42:43,443 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) [4a8fa0db] Principal admin::internal synchronized >2014-09-18 12:42:45,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-09-18 13:09:07,765 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2014-09-18 13:09:07,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-18 13:09:08,130 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-09-18 13:09:32,548 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-30) [4a6cbf91] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 1e37baca-9686-4106-8f2b-a3c35afb694f Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-18 13:09:32,620 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4a6cbf91] START, SetVdsStatusVDSCommand(HostName = rose07, HostId = 1e37baca-9686-4106-8f2b-a3c35afb694f, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 657d3b6c >2014-09-18 13:09:32,666 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [4a6cbf91] FINISH, SetVdsStatusVDSCommand, log id: 657d3b6c >2014-09-18 13:09:32,910 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-30) [4a6cbf91] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 1e37baca-9686-4106-8f2b-a3c35afb694f Type: VDS >2014-09-18 13:09:32,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [4a6cbf91] Correlation ID: 4a6cbf91, Job ID: faed72df-724d-4455-ae16-87b61f9c642e, Call Stack: null, Custom Event ID: -1, Message: Host rose07 was switched to Maintenance mode by admin. >2014-09-18 13:09:35,128 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 1e37baca-9686-4106-8f2b-a3c35afb694f : rose07 >2014-09-18 13:09:35,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-8) Clearing cache of pool: 861a69f1-0ce2-406b-90d2-64a39fbe1b14 for problematic entities of VDS: rose07. >2014-09-18 13:09:35,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-8) Removing vds [1e37baca-9686-4106-8f2b-a3c35afb694f] from the domain in maintenance cache >2014-09-18 13:09:35,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-8) Removing host(s) [1e37baca-9686-4106-8f2b-a3c35afb694f] from hosts unseen domain report cache >2014-09-18 13:09:40,892 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-37) [6299333e] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7fa33be7-7aff-42a9-9e59-81589d67380d Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-18 13:09:40,897 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) [6299333e] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 7fa33be7-7aff-42a9-9e59-81589d67380d, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 7282189b >2014-09-18 13:09:40,914 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) [6299333e] FINISH, SetVdsStatusVDSCommand, log id: 7282189b >2014-09-18 13:09:41,049 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-37) [6299333e] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7fa33be7-7aff-42a9-9e59-81589d67380d Type: VDS >2014-09-18 13:09:41,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [6299333e] Correlation ID: 6299333e, Job ID: 8d74fbd8-aee5-4727-8fe2-517d66e6d045, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was switched to Maintenance mode by admin. >2014-09-18 13:09:43,537 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 7fa33be7-7aff-42a9-9e59-81589d67380d : monique-vds05.tlv.redhat.com >2014-09-18 13:09:43,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-21) Clearing cache of pool: 861a69f1-0ce2-406b-90d2-64a39fbe1b14 for problematic entities of VDS: monique-vds05.tlv.redhat.com. >2014-09-18 13:09:43,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-21) Removing vds [7fa33be7-7aff-42a9-9e59-81589d67380d] from the domain in maintenance cache >2014-09-18 13:09:43,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-21) Removing host(s) [7fa33be7-7aff-42a9-9e59-81589d67380d] from hosts unseen domain report cache >2014-09-18 13:42:43,455 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-18 13:42:45,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-09-18 14:42:43,464 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-09-18 14:42:45,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-09-18 15:20:59,042 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) [1cdff701] Lock Acquired to object EngineLock [exclusiveLocks= key: 1e37baca-9686-4106-8f2b-a3c35afb694f value: VDS >, sharedLocks= ] >2014-09-18 15:20:59,124 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-3) [1cdff701] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 1e37baca-9686-4106-8f2b-a3c35afb694f Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-18 15:20:59,192 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-3) [1cdff701] START, RemoveVdsVDSCommand( HostId = 1e37baca-9686-4106-8f2b-a3c35afb694f), log id: 4240f7f9 >2014-09-18 15:20:59,193 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-3) [1cdff701] vdsManager::disposing >2014-09-18 15:20:59,193 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-3) [1cdff701] FINISH, RemoveVdsVDSCommand, log id: 4240f7f9 >2014-09-18 15:20:59,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [1cdff701] Correlation ID: 1cdff701, Call Stack: null, Custom Event ID: -1, Message: Host rose07 was removed by admin. >2014-09-18 15:20:59,202 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-3) [1cdff701] Lock freed to object EngineLock [exclusiveLocks= key: 1e37baca-9686-4106-8f2b-a3c35afb694f value: VDS >, sharedLocks= ] >2014-09-18 15:21:04,791 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) [3b9d7e91] Lock Acquired to object EngineLock [exclusiveLocks= key: 7fa33be7-7aff-42a9-9e59-81589d67380d value: VDS >, sharedLocks= ] >2014-09-18 15:21:04,818 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 7fa33be7-7aff-42a9-9e59-81589d67380d Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-18 15:21:04,903 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] START, RemoveVdsVDSCommand( HostId = 7fa33be7-7aff-42a9-9e59-81589d67380d), log id: 1207482b >2014-09-18 15:21:04,903 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] vdsManager::disposing >2014-09-18 15:21:04,903 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] FINISH, RemoveVdsVDSCommand, log id: 1207482b >2014-09-18 15:21:04,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] Correlation ID: 3b9d7e91, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was removed by admin. >2014-09-18 15:21:04,910 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-18) [3b9d7e91] Lock freed to object EngineLock [exclusiveLocks= key: 7fa33be7-7aff-42a9-9e59-81589d67380d value: VDS >, sharedLocks= ] >2014-09-18 15:27:57,271 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-11) [7a421184] Running command: AddVdsCommand internal: false. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-18 15:27:57,404 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-18 15:27:57,416 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Lock freed to object EngineLock [exclusiveLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: REGISTER_VDS >, sharedLocks= ] >2014-09-18 15:27:57,420 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] START, RemoveVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 7ae25a1a >2014-09-18 15:27:57,420 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] FINISH, RemoveVdsVDSCommand, log id: 7ae25a1a >2014-09-18 15:27:57,436 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] START, AddVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 161a60d9 >2014-09-18 15:27:57,436 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] AddVds - entered , starting logic to add VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 >2014-09-18 15:27:57,439 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] AddVds - VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 was added, will try to add it to the resource manager >2014-09-18 15:27:57,439 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Entered VdsManager constructor >2014-09-18 15:27:57,452 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-09-18 15:27:57,454 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-11) [25e1c4b5] VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 was added to the Resource Manager >2014-09-18 15:27:57,454 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [25e1c4b5] FINISH, AddVdsVDSCommand, log id: 161a60d9 >2014-09-18 15:27:57,468 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-18 15:27:57,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [25e1c4b5] Correlation ID: 7a421184, Job ID: cd2558e1-5180-48dd-8505-d6bdf518c3ed, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 was added by admin. >2014-09-18 15:27:57,589 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Lock Acquired to object EngineLock [exclusiveLocks= key: eec426a9-0846-4c02-83f0-c6fdb9566de0 value: VDS >, sharedLocks= ] >2014-09-18 15:27:57,643 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-18 15:27:57,651 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Before Installation host eec426a9-0846-4c02-83f0-c6fdb9566de0, monique-vds05 >2014-09-18 15:27:57,689 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] START, SetVdsStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 56651857 >2014-09-18 15:27:57,709 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] FINISH, SetVdsStatusVDSCommand, log id: 56651857 >2014-09-18 15:27:57,816 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Installation monique-vds05.tlv.redhat.com: Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: c8:8c:ff:bf:60:67:ae:6a:32:6a:b5:32:62:69:d8:30 >2014-09-18 15:27:57,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: c8:8c:ff:bf:60:67:ae:6a:32:6a:b5:32:62:69:d8:30. >2014-09-18 15:27:57,891 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Installation of monique-vds05.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-18 15:27:57,911 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-6-thread-48) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is out of date, re-creating >2014-09-18 15:27:58,235 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-48) SSH execute root@monique-vds05.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-18 15:27:59,103 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Initializing >2014-09-18 15:27:59,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Initializing. >2014-09-18 15:27:59,179 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment setup >2014-09-18 15:27:59,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment setup. >2014-09-18 15:27:59,232 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment packages setup >2014-09-18 15:27:59,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment packages setup. >2014-09-18 15:28:00,457 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Initializing Host: monique-vds05 >2014-09-18 15:28:04,317 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Programs detection >2014-09-18 15:28:04,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Programs detection. >2014-09-18 15:28:04,331 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment customization >2014-09-18 15:28:04,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment customization. >2014-09-18 15:28:04,581 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Kdump unsupported >2014-09-18 15:28:04,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Kdump unsupported. >2014-09-18 15:28:04,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140918152813-rmqwxq.log' >2014-09-18 15:28:04,626 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Logs at host located at: '/tmp/ovirt-host-deploy-20140918152813-rmqwxq.log'. >2014-09-18 15:28:04,854 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds05.tlv.redhat.com reports unique id 4C4C4544-0032-4310-8051-B7C04F39354A >2014-09-18 15:28:04,919 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0032-4310-8051-B7C04F39354A to Host monique-vds05.tlv.redhat.com >2014-09-18 15:28:05,373 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Disabling Kdump integration >2014-09-18 15:28:05,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Disabling Kdump integration. >2014-09-18 15:28:05,550 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Setup validation >2014-09-18 15:28:05,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Setup validation. >2014-09-18 15:28:05,660 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Hardware supports virtualization >2014-09-18 15:28:05,690 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Hardware supports virtualization. >2014-09-18 15:28:05,748 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction setup >2014-09-18 15:28:05,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Transaction setup. >2014-09-18 15:28:05,768 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-09-18 15:28:05,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Misc configuration. >2014-09-18 15:28:05,799 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Package installation >2014-09-18 15:28:05,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Package installation. >2014-09-18 15:28:14,827 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-09-18 15:28:14,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Misc configuration. >2014-09-18 15:28:25,547 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Enrolling certificate >2014-09-18 15:28:25,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Enrolling certificate. >2014-09-18 15:28:26,796 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction commit >2014-09-18 15:28:26,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Transaction commit. >2014-09-18 15:28:26,821 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Closing up >2014-09-18 15:28:26,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Closing up. >2014-09-18 15:28:27,944 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stopping libvirtd >2014-09-18 15:28:27,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stopping libvirtd. >2014-09-18 15:28:28,004 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Starting vdsm >2014-09-18 15:28:28,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Starting vdsm. >2014-09-18 15:28:31,856 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Pre-termination >2014-09-18 15:28:31,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Pre-termination. >2014-09-18 15:28:31,888 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140918152831-monique-vds05.tlv.redhat.com-2090b7c4.log' >2014-09-18 15:28:31,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140918152831-monique-vds05.tlv.redhat.com-2090b7c4.log'. >2014-09-18 15:28:32,109 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Termination >2014-09-18 15:28:32,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2090b7c4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Termination. >2014-09-18 15:28:32,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-09-18 15:28:32,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Command PollVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-09-18 15:28:32,769 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Engine managed to communicate with VDSM agent on host monique-vds05 >2014-09-18 15:28:33,128 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] The management network rhevm is already configured on host monique-vds05 >2014-09-18 15:28:33,130 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] START, SetVdsStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7e65d447 >2014-09-18 15:28:33,146 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] FINISH, SetVdsStatusVDSCommand, log id: 7e65d447 >2014-09-18 15:28:33,146 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] After Installation host monique-vds05, VDS >2014-09-18 15:28:33,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Correlation ID: 2090b7c4, Job ID: cd2558e1-5180-48dd-8505-d6bdf518c3ed, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 installed >2014-09-18 15:28:33,167 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-48) [2090b7c4] Lock freed to object EngineLock [exclusiveLocks= key: eec426a9-0846-4c02-83f0-c6fdb9566de0 value: VDS >, sharedLocks= ] >2014-09-18 15:28:33,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 227b06fe >2014-09-18 15:28:33,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, GetHardwareInfoVDSCommand, log id: 227b06fe >2014-09-18 15:28:33,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [95464a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-18 15:28:33,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [95464a6] START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 251801ba >2014-09-18 15:28:33,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [95464a6] FINISH, GetHardwareInfoVDSCommand, log id: 251801ba >2014-09-18 15:28:33,926 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-75) [48a28030] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:28:33,937 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:28:33,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] START, ConnectStorageServerVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 163e45e >2014-09-18 15:28:34,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 163e45e >2014-09-18 15:28:34,626 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] Host monique-vds05 storage connection was succeeded >2014-09-18 15:28:34,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vdsId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 4128c59c >2014-09-18 15:28:34,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 4128c59c >2014-09-18 15:28:34,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 7bc4c733 >2014-09-18 15:28:34,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-75) [53acdcff] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7bc4c733 >2014-09-18 15:28:34,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [53acdcff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05 was set to Up. >2014-09-18 15:28:34,978 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [53acdcff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-18 15:28:34,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [53acdcff] Correlation ID: 48a28030, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-18 15:28:35,006 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [15d74ab4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-18 15:28:35,013 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-18 15:29:54,483 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-35) Command GetStatsVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond >2014-09-18 15:29:54,486 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:29:54,504 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:29:54,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) vds::refreshVdsStats Failed getVdsStats, vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, error = VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond >2014-09-18 15:29:54,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:680) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:634) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1976) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:29:54,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [4a8fa0db] Command SpmStatusVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-09-18 15:29:54,523 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:29:54,530 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-18 15:29:54,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond >2014-09-18 15:29:54,560 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Correlation ID: e92f6ca, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host moniqu01, Error: Network error during communication with the Host.). >2014-09-18 15:29:54,674 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-09-18 15:29:54,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Command SpmStatusVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:29:54,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Irs placed on server 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f failed. Proceed Failover >2014-09-18 15:29:54,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) [e92f6ca] hostFromVds::selectedVds - monique-vds05, spmStatus Free, storage pool DC1 >2014-09-18 15:29:54,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) [e92f6ca] starting spm on vds monique-vds05, storage pool DC1, prevId -1, LVER -1 >2014-09-18 15:29:54,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] START, SpmStartVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47ee9dca >2014-09-18 15:29:54,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] spmStart polling started: taskId = 8b0a2444-280c-4c48-a3e7-c468a9acb698 >2014-09-18 15:29:57,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:29:57,551 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:29:57,558 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:29:57,586 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:00,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:30:00,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:30:00,605 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:30:00,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:03,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [76aaf268] Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:30:03,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [76aaf268] Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:30:03,631 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [76aaf268] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:30:03,638 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [76aaf268] Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:06,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:30:06,648 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:30:06,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:30:06,660 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:09,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:30:09,671 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:30:09,676 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:30:09,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:12,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-18 15:30:12,692 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host moniqu01 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-18 15:30:12,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-18 15:30:12,704 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-18 15:30:16,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [2cbdcee] START, GetHardwareInfoVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vds=Host[moniqu01,9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f]), log id: 14958834 >2014-09-18 15:30:16,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [2cbdcee] FINISH, GetHardwareInfoVDSCommand, log id: 14958834 >2014-09-18 15:30:16,444 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [5ea3bc7a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f Type: VDS >2014-09-18 15:30:16,482 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [5c1871b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:30:16,516 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:30:16,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] START, ConnectStorageServerVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 21da9652 >2014-09-18 15:30:16,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 21da9652 >2014-09-18 15:30:16,638 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] Host moniqu01 storage connection was succeeded >2014-09-18 15:30:16,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) START, ConnectStoragePoolVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, vdsId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 41840020 >2014-09-18 15:30:17,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 41840020 >2014-09-18 15:30:42,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] START, SetMOMPolicyParametersVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f), log id: 4f35f566 >2014-09-18 15:30:43,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c19808c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4f35f566 >2014-09-18 15:30:43,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5c19808c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host moniqu01 was set to Up. >2014-09-18 15:30:43,103 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5c19808c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host moniqu01. >2014-09-18 15:30:43,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5c19808c] Correlation ID: 5c1871b, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-18 15:30:43,157 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [14e59306] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f Type: VDS >2014-09-18 15:31:16,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] spmStart polling ended: taskId = 8b0a2444-280c-4c48-a3e7-c468a9acb698 task status = finished >2014-09-18 15:31:17,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] spmStart polling ended, spm status: SPM >2014-09-18 15:31:17,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] START, HSMClearTaskVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, taskId=8b0a2444-280c-4c48-a3e7-c468a9acb698), log id: 315107a >2014-09-18 15:31:17,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] FINISH, HSMClearTaskVDSCommand, log id: 315107a >2014-09-18 15:31:17,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) [e92f6ca] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1405892c, log id: 47ee9dca >2014-09-18 15:31:17,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-09-18 15:31:17,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05 (Address: monique-vds05.tlv.redhat.com). >2014-09-18 15:31:17,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: 1dad289f >2014-09-18 15:31:17,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-18 15:31:17,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 4617e655 >2014-09-18 15:31:17,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4617e655 >2014-09-18 15:31:17,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1dad289f >2014-09-18 15:31:17,460 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-48) [e92f6ca] Discovered no tasks on Storage Pool DC1 >2014-09-18 15:34:11,653 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-18 15:34:11,656 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] START, SetVdsStatusVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 11c866d0 >2014-09-18 15:34:11,670 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] FINISH, SetVdsStatusVDSCommand, log id: 11c866d0 >2014-09-18 15:34:11,829 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f Type: VDS >2014-09-18 15:34:11,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] START, SetHaMaintenanceModeVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f), log id: 3b606d3d >2014-09-18 15:34:11,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3b606d3d >2014-09-18 15:34:11,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [106ef8cb] Correlation ID: 106ef8cb, Job ID: 4d19af3b-7ebd-4617-8113-b8bc049b9938, Call Stack: null, Custom Event ID: -1, Message: Host moniqu01 was switched to Maintenance mode by admin. >2014-09-18 15:34:39,400 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f : moniqu01 >2014-09-18 15:34:39,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-42) Clearing cache of pool: 43e1738a-3a93-481a-befe-65c12fbc155f for problematic entities of VDS: moniqu01. >2014-09-18 15:34:39,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-42) Removing vds [9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f] from the domain in maintenance cache >2014-09-18 15:34:39,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-42) Removing host(s) [9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f] from hosts unseen domain report cache >2014-09-18 15:34:39,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, DisconnectStoragePoolVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, vds_spm_id = 1), log id: 57369979 >2014-09-18 15:34:43,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, DisconnectStoragePoolVDSCommand, log id: 57369979 >2014-09-18 15:34:43,069 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [333b6810] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-18 15:34:43,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [333b6810] START, DisconnectStorageServerVDSCommand(HostName = moniqu01, HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b6f34a6 >2014-09-18 15:34:43,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [333b6810] FINISH, DisconnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 7b6f34a6 >2014-09-18 15:34:45,491 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) [60c34acc] Lock Acquired to object EngineLock [exclusiveLocks= key: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f value: VDS >, sharedLocks= ] >2014-09-18 15:34:45,543 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-10) [60c34acc] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-18 15:34:45,618 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-10) [60c34acc] START, RemoveVdsVDSCommand( HostId = 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f), log id: cb07ee7 >2014-09-18 15:34:45,618 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-10) [60c34acc] vdsManager::disposing >2014-09-18 15:34:45,618 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-10) [60c34acc] FINISH, RemoveVdsVDSCommand, log id: cb07ee7 >2014-09-18 15:34:45,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [60c34acc] Correlation ID: 60c34acc, Call Stack: null, Custom Event ID: -1, Message: Host moniqu01 was removed by admin. >2014-09-18 15:34:45,637 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-10) [60c34acc] Lock freed to object EngineLock [exclusiveLocks= key: 9c18cc8b-15e8-476e-bb4e-6fc39bd3eb4f value: VDS >, sharedLocks= ] >2014-09-18 15:42:43,475 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Principal admin::internal synchronized >2014-09-18 15:42:45,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-18 16:42:43,488 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-09-18 16:42:45,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-09-18 17:42:43,501 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-09-18 17:42:45,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-09-18 18:42:43,517 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-39) Principal admin::internal synchronized >2014-09-18 18:42:45,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-09-18 19:42:43,526 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) Principal admin::internal synchronized >2014-09-18 19:42:45,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-09-18 20:42:43,546 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) [606158b9] Principal admin::internal synchronized >2014-09-18 20:42:45,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-18 21:42:43,576 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-38) Principal admin::internal synchronized >2014-09-18 21:42:45,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-09-18 22:42:43,586 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [14e59306] Principal admin::internal synchronized >2014-09-18 22:42:45,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-18 23:42:43,598 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-30) Principal admin::internal synchronized >2014-09-18 23:42:45,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-09-19 00:42:43,606 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-19 00:42:45,835 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-09-19 01:42:43,614 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-09-19 01:42:45,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-09-19 02:42:43,626 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-19 02:42:45,873 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) [325ae059] Attempting to update VMs/Templates Ovf. >2014-09-19 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-23) Start deleteAgedOutAuditLogs >2014-09-19 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Start deleteAgedOutCommandEntities >2014-09-19 03:35:35,012 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-23) Finished deleteAgedOutAuditLogs >2014-09-19 03:35:35,013 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Finished deleteAgedOutCommandEntities >2014-09-19 03:42:43,638 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-19 03:42:45,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-09-19 04:42:43,657 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) [184f5e9a] Principal admin::internal synchronized >2014-09-19 04:42:45,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-09-19 05:42:43,665 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-09-19 05:42:45,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-09-19 06:42:43,675 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-09-19 06:42:45,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-09-19 07:42:43,684 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-09-19 07:42:45,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-09-19 08:42:43,691 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [632768a2] Principal admin::internal synchronized >2014-09-19 08:42:45,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [164567a] Attempting to update VMs/Templates Ovf. >2014-09-19 09:42:43,700 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized >2014-09-19 09:42:46,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-19 10:19:33,545 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) Command ListVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-09-19 10:19:33,546 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-19 10:19:33,867 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-19 10:19:33,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:413) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:982) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:931) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy97.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-19 10:19:33,933 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Failed to refresh VDS , vds = eec426a9-0846-4c02-83f0-c6fdb9566de0 : monique-vds05, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-09-19 10:19:55,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) Command SpmStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-09-19 10:19:55,972 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [13609e21] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-19 10:19:55,979 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) [13609e21] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-19 10:19:56,013 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [13609e21] Correlation ID: 13609e21, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds05, Error: Network error during communication with the Host.). >2014-09-19 10:19:56,071 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [13609e21] IrsBroker::Failed::GetStoragePoolInfoVDS due to: SocketException: Connection reset >2014-09-19 10:19:56,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 3e155c03 >2014-09-19 10:19:56,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 3e155c03 >2014-09-19 10:19:56,451 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [37cbfb65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-19 10:19:56,496 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [147ee6bf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-19 10:19:56,524 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-19 10:19:56,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] START, ConnectStorageServerVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2ea0b68a >2014-09-19 10:19:56,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 2ea0b68a >2014-09-19 10:19:56,683 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] Host monique-vds05 storage connection was succeeded >2014-09-19 10:19:56,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vdsId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 58ff2759 >2014-09-19 10:19:56,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 58ff2759 >2014-09-19 10:19:57,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 6e3a997b >2014-09-19 10:19:57,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6e3a997b >2014-09-19 10:19:57,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05 was set to Up. >2014-09-19 10:19:57,313 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-19 10:19:57,345 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [1c0a9cb6] Correlation ID: 147ee6bf, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-19 10:19:57,381 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [4ab5d8c5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-19 10:42:43,712 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized >2014-09-19 10:42:46,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-09-19 11:42:43,736 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-19 11:42:46,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-09-19 12:42:43,759 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-09-19 12:42:46,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-09-19 13:42:43,774 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-09-19 13:42:46,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-09-19 14:42:43,787 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-09-19 14:42:46,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-09-19 15:42:43,796 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-38) Principal admin::internal synchronized >2014-09-19 15:42:46,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-09-19 16:42:43,817 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) [13609e21] Principal admin::internal synchronized >2014-09-19 16:42:46,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-09-19 17:42:43,831 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-19 17:42:46,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-19 18:42:43,853 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-09-19 18:42:46,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-09-19 19:42:43,861 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-19 19:42:46,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-09-19 20:42:43,872 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) [164567a] Principal admin::internal synchronized >2014-09-19 20:42:46,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-09-19 21:42:43,884 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [14e59306] Principal admin::internal synchronized >2014-09-19 21:42:46,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-09-19 22:42:43,892 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-09-19 22:42:46,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-09-19 23:42:43,901 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-09-19 23:42:46,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-09-20 00:42:43,910 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-09-20 00:42:46,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-09-20 01:42:43,919 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-91) Principal admin::internal synchronized >2014-09-20 01:42:46,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) [76aaf268] Attempting to update VMs/Templates Ovf. >2014-09-20 02:42:43,939 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-09-20 02:42:46,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-20 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Start deleteAgedOutAuditLogs >2014-09-20 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-63) Start deleteAgedOutCommandEntities >2014-09-20 03:35:35,015 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Finished deleteAgedOutAuditLogs >2014-09-20 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-63) Finished deleteAgedOutCommandEntities >2014-09-20 03:42:43,951 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized >2014-09-20 03:42:46,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-09-20 04:42:43,978 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-09-20 04:42:46,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-09-20 05:42:43,994 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) [78d7871d] Principal admin::internal synchronized >2014-09-20 05:42:46,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-09-20 06:42:44,019 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) Principal admin::internal synchronized >2014-09-20 06:42:46,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-09-20 07:42:44,042 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-20 07:42:46,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) [606158b9] Attempting to update VMs/Templates Ovf. >2014-09-20 08:42:44,060 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-20 08:42:46,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-20 09:42:44,068 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Principal admin::internal synchronized >2014-09-20 09:42:46,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-09-20 10:42:44,095 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-09-20 10:42:46,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-09-20 11:42:44,108 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-09-20 11:42:46,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-09-20 12:42:44,116 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-09-20 12:42:46,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-09-20 13:42:44,124 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-09-20 13:42:46,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-09-20 14:42:44,135 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized >2014-09-20 14:42:46,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-09-20 15:42:44,145 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-20 15:42:46,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-09-20 16:42:44,156 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-20 16:42:46,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-09-20 17:42:44,167 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-09-20 17:42:46,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-09-20 18:42:44,175 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-20 18:42:46,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-09-20 19:42:44,191 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-09-20 19:42:46,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-09-20 20:42:44,200 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-09-20 20:42:46,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-09-20 21:42:44,209 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-09-20 21:42:46,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-09-20 22:42:44,220 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-09-20 22:42:46,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-20 23:42:44,228 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-09-20 23:42:47,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-09-21 00:42:44,237 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-09-21 00:42:47,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-09-21 01:42:44,246 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-09-21 01:42:47,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-09-21 02:42:44,257 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) [184f5e9a] Principal admin::internal synchronized >2014-09-21 02:42:47,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-09-21 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Start deleteAgedOutAuditLogs >2014-09-21 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) Start deleteAgedOutCommandEntities >2014-09-21 03:35:35,013 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Finished deleteAgedOutAuditLogs >2014-09-21 03:35:35,016 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) Finished deleteAgedOutCommandEntities >2014-09-21 03:42:44,273 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-89) Principal admin::internal synchronized >2014-09-21 03:42:47,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [632768a2] Attempting to update VMs/Templates Ovf. >2014-09-21 04:42:44,282 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-09-21 04:42:47,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-09-21 05:42:44,296 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-09-21 05:42:47,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-09-21 06:42:44,307 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-32) Principal admin::internal synchronized >2014-09-21 06:42:47,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-09-21 07:42:44,332 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-31) [325ae059] Principal admin::internal synchronized >2014-09-21 07:42:47,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-09-21 08:42:44,341 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) [37ae315a] Principal admin::internal synchronized >2014-09-21 08:42:47,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-09-21 09:42:44,352 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) [164567a] Principal admin::internal synchronized >2014-09-21 09:42:47,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-09-21 10:42:44,363 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-09-21 10:42:47,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-09-21 11:42:44,372 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-38) Principal admin::internal synchronized >2014-09-21 11:42:47,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-21 12:42:44,390 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-21 12:42:47,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-09-21 13:42:44,399 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-09-21 13:42:47,300 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-09-21 14:42:44,407 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-09-21 14:42:47,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [14e59306] Attempting to update VMs/Templates Ovf. >2014-09-21 15:42:44,417 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-09-21 15:42:47,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-09-21 16:42:44,428 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-09-21 16:42:47,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-21 17:42:44,438 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-21 17:42:47,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-09-21 18:42:44,458 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-21 18:42:47,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-09-21 19:42:44,470 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-09-21 19:42:47,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-09-21 20:42:44,494 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-09-21 20:42:47,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-09-21 21:42:44,511 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-09-21 21:42:47,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-21 22:42:44,522 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-21 22:42:47,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-09-21 23:42:44,531 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-09-21 23:42:47,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-22 00:42:44,545 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-89) Principal admin::internal synchronized >2014-09-22 00:42:47,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-09-22 01:42:44,557 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Principal admin::internal synchronized >2014-09-22 01:42:47,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-09-22 02:42:44,565 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-09-22 02:42:47,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-09-22 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) Start deleteAgedOutAuditLogs >2014-09-22 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) Start deleteAgedOutCommandEntities >2014-09-22 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) Finished deleteAgedOutAuditLogs >2014-09-22 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) Finished deleteAgedOutCommandEntities >2014-09-22 03:42:44,575 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-22 03:42:47,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-09-22 04:42:44,586 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-22 04:42:47,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-09-22 05:42:44,597 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) [13609e21] Principal admin::internal synchronized >2014-09-22 05:42:47,633 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-09-22 06:42:44,608 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Principal admin::internal synchronized >2014-09-22 06:42:47,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-09-22 07:42:44,616 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-09-22 07:42:47,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-09-22 08:42:44,625 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [333b6810] Principal admin::internal synchronized >2014-09-22 08:42:47,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-09-22 09:42:44,633 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-22 09:42:47,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-09-22 09:49:02,283 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-09-22 09:49:02,283 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-09-22 10:01:45,149 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginAdminUserCommand internal: false. >2014-09-22 10:01:45,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-22 10:01:45,364 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-09-22 10:42:44,642 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) [164567a] Principal admin::internal synchronized >2014-09-22 10:42:47,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) [4ab5d8c5] Attempting to update VMs/Templates Ovf. >2014-09-22 11:42:44,653 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-100) Principal admin::internal synchronized >2014-09-22 11:42:47,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-22 12:42:44,661 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-09-22 12:42:47,802 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-22 13:42:44,670 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) Principal admin::internal synchronized >2014-09-22 13:42:47,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-09-22 14:42:44,679 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-22 14:42:47,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-09-22 15:42:44,688 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-09-22 15:42:47,859 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-09-22 16:07:11,314 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-4) [1be463fc] Running command: AddVdsCommand internal: false. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-22 16:07:11,377 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [95cacae] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:11,393 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [95cacae] Lock freed to object EngineLock [exclusiveLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: REGISTER_VDS >, sharedLocks= ] >2014-09-22 16:07:11,395 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] START, RemoveVdsVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471), log id: 5818cb1c >2014-09-22 16:07:11,396 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] FINISH, RemoveVdsVDSCommand, log id: 5818cb1c >2014-09-22 16:07:11,398 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] START, AddVdsVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471), log id: 38ca7a0f >2014-09-22 16:07:11,398 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] AddVds - entered , starting logic to add VDS 9714e5d7-b19b-4335-9d96-d31983c4a471 >2014-09-22 16:07:11,400 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] AddVds - VDS 9714e5d7-b19b-4335-9d96-d31983c4a471 was added, will try to add it to the resource manager >2014-09-22 16:07:11,400 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [95cacae] Entered VdsManager constructor >2014-09-22 16:07:11,412 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [95cacae] Initialize vdsBroker (aqua-vds3.qa.lab.tlv.redhat.com,54,321) >2014-09-22 16:07:11,413 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [95cacae] VDS 9714e5d7-b19b-4335-9d96-d31983c4a471 was added to the Resource Manager >2014-09-22 16:07:11,413 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [95cacae] FINISH, AddVdsVDSCommand, log id: 38ca7a0f >2014-09-22 16:07:11,426 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [95cacae] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host aqua-vds3.qa.lab.tlv.redhat.com. >2014-09-22 16:07:11,476 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Lock Acquired to object EngineLock [exclusiveLocks= key: 9714e5d7-b19b-4335-9d96-d31983c4a471 value: VDS >, sharedLocks= ] >2014-09-22 16:07:11,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [95cacae] Correlation ID: 1be463fc, Job ID: 8f74e6ff-5326-44ef-90bc-fbbb0fbab833, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com was added by admin. >2014-09-22 16:07:11,494 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:11,494 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Before Installation host 9714e5d7-b19b-4335-9d96-d31983c4a471, aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:07:11,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] START, SetVdsStatusVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5b6af8b >2014-09-22 16:07:11,506 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] FINISH, SetVdsStatusVDSCommand, log id: 5b6af8b >2014-09-22 16:07:11,538 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Installation aqua-vds3.qa.lab.tlv.redhat.com: Connected to host aqua-vds3.qa.lab.tlv.redhat.com with SSH key fingerprint: 08:cc:a4:c4:7e:5e:ac:b8:bd:b0:56:c4:68:56:49:36 >2014-09-22 16:07:11,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Connected to host aqua-vds3.qa.lab.tlv.redhat.com with SSH key fingerprint: 08:cc:a4:c4:7e:5e:ac:b8:bd:b0:56:c4:68:56:49:36. >2014-09-22 16:07:11,629 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Installation of aqua-vds3.qa.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-22 16:07:11,679 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-42) SSH execute root@aqua-vds3.qa.lab.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-22 16:07:12,481 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Initializing >2014-09-22 16:07:12,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Initializing. >2014-09-22 16:07:12,524 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Environment setup >2014-09-22 16:07:12,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Environment setup. >2014-09-22 16:07:12,566 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Environment packages setup >2014-09-22 16:07:12,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Environment packages setup. >2014-09-22 16:07:14,417 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Initializing Host: aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:07:17,621 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Programs detection >2014-09-22 16:07:17,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Programs detection. >2014-09-22 16:07:17,634 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Environment customization >2014-09-22 16:07:17,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Environment customization. >2014-09-22 16:07:17,743 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Kdump unsupported >2014-09-22 16:07:17,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Kdump unsupported. >2014-09-22 16:07:17,812 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140922161612-searb6.log' >2014-09-22 16:07:17,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20140922161612-searb6.log'. >2014-09-22 16:07:17,915 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host aqua-vds3.qa.lab.tlv.redhat.com reports unique id e87fb74b-a809-4ce1-b6a1-60ecd4bd87de >2014-09-22 16:07:17,917 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id e87fb74b-a809-4ce1-b6a1-60ecd4bd87de to Host aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:07:18,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Disabling Kdump integration >2014-09-22 16:07:18,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Disabling Kdump integration. >2014-09-22 16:07:18,266 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Setup validation >2014-09-22 16:07:18,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Setup validation. >2014-09-22 16:07:18,292 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Hardware supports virtualization >2014-09-22 16:07:18,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Hardware supports virtualization. >2014-09-22 16:07:18,378 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Transaction setup >2014-09-22 16:07:18,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Transaction setup. >2014-09-22 16:07:18,550 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-09-22 16:07:18,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-09-22 16:07:18,560 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Package installation >2014-09-22 16:07:18,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Package installation. >2014-09-22 16:07:19,173 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-09-22 16:07:19,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-09-22 16:07:20,159 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Enrolling certificate >2014-09-22 16:07:20,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Enrolling certificate. >2014-09-22 16:07:21,707 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Transaction commit >2014-09-22 16:07:21,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Transaction commit. >2014-09-22 16:07:21,734 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Closing up >2014-09-22 16:07:21,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Closing up. >2014-09-22 16:07:22,089 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Restarting nrpe service >2014-09-22 16:07:22,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Restarting nrpe service. >2014-09-22 16:07:22,946 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stopping libvirtd >2014-09-22 16:07:22,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stopping libvirtd. >2014-09-22 16:07:23,361 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Starting vdsm >2014-09-22 16:07:23,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Starting vdsm. >2014-09-22 16:07:24,860 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Pre-termination >2014-09-22 16:07:24,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Pre-termination. >2014-09-22 16:07:24,898 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140922160724-aqua-vds3.qa.lab.tlv.redhat.com-2ab2222c.log' >2014-09-22 16:07:24,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140922160724-aqua-vds3.qa.lab.tlv.redhat.com-2ab2222c.log'. >2014-09-22 16:07:25,131 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation aqua-vds3.qa.lab.tlv.redhat.com: Stage: Termination >2014-09-22 16:07:25,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2ab2222c, Call Stack: null, Custom Event ID: -1, Message: Installing Host aqua-vds3.qa.lab.tlv.redhat.com. Stage: Termination. >2014-09-22 16:07:25,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-09-22 16:07:25,299 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Command PollVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-09-22 16:07:25,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-09-22 16:07:25,895 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Command PollVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-09-22 16:07:26,423 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Engine managed to communicate with VDSM agent on host aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:07:26,750 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] The management network rhevm is already configured on host aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:07:26,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] START, SetVdsStatusVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 295f6a99 >2014-09-22 16:07:26,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] FINISH, SetVdsStatusVDSCommand, log id: 295f6a99 >2014-09-22 16:07:26,780 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] After Installation host aqua-vds3.qa.lab.tlv.redhat.com, VDS >2014-09-22 16:07:26,804 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Correlation ID: 2ab2222c, Job ID: 8f74e6ff-5326-44ef-90bc-fbbb0fbab833, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com installed >2014-09-22 16:07:26,813 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-42) [2ab2222c] Lock freed to object EngineLock [exclusiveLocks= key: 9714e5d7-b19b-4335-9d96-d31983c4a471 value: VDS >, sharedLocks= ] >2014-09-22 16:07:29,590 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com does not enforce SELinux. >2014-09-22 16:07:29,590 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host aqua-vds3.qa.lab.tlv.redhat.com does not report SELinux enforcement information. >2014-09-22 16:07:29,646 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [1e252e1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:29,802 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [1e252e1f] Host aqua-vds3.qa.lab.tlv.redhat.com does not report SELinux enforcement information. >2014-09-22 16:07:29,896 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [115df546] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-22 16:07:29,908 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-22 16:07:29,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] START, ConnectStorageServerVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: c258636 >2014-09-22 16:07:30,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: c258636 >2014-09-22 16:07:30,245 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] Host aqua-vds3.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-22 16:07:30,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) START, ConnectStoragePoolVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, vdsId = 9714e5d7-b19b-4335-9d96-d31983c4a471, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 1f5084e8 >2014-09-22 16:07:30,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) FINISH, ConnectStoragePoolVDSCommand, log id: 1f5084e8 >2014-09-22 16:07:30,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] START, SetMOMPolicyParametersVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471), log id: 6e235749 >2014-09-22 16:07:30,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6e235749 >2014-09-22 16:07:30,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host aqua-vds3.qa.lab.tlv.redhat.com was set to Up. >2014-09-22 16:07:30,911 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host aqua-vds3.qa.lab.tlv.redhat.com. >2014-09-22 16:07:30,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3f7d7e2e] Correlation ID: 115df546, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-22 16:07:30,972 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [68a22de2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:30,981 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [362e588] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:31,083 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [6293e326] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:07:31,087 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [6293e326] START, SetVdsStatusVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, status=NonOperational, nonOperationalReason=CLUSTER_VERSION_INCOMPATIBLE_WITH_CLUSTER, stopSpmFailureLogged=false), log id: 6a814ee0 >2014-09-22 16:07:31,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [6293e326] FINISH, SetVdsStatusVDSCommand, log id: 6a814ee0 >2014-09-22 16:07:31,160 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6293e326] Correlation ID: 6293e326, Job ID: 6d098da4-014a-416c-8819-58de95b8107c, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com is compatible with versions (3.0,3.1,3.2,3.3,3.4) and cannot join Cluster cluster1 which is set to version 3.5. >2014-09-22 16:07:31,173 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [6293e326] Host 9714e5d7-b19b-4335-9d96-d31983c4a471 : aqua-vds3.qa.lab.tlv.redhat.com is already in NonOperational status for reason CLUSTER_VERSION_INCOMPATIBLE_WITH_CLUSTER. SetNonOperationalVds command is skipped. >2014-09-22 16:42:44,697 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-09-22 16:42:47,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [e92f6ca] Attempting to update VMs/Templates Ovf. >2014-09-22 16:43:00,147 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-8) [69585cfd] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-22 16:43:00,151 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-8) [69585cfd] START, SetVdsStatusVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 10bcd0d6 >2014-09-22 16:43:00,165 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-8) [69585cfd] FINISH, SetVdsStatusVDSCommand, log id: 10bcd0d6 >2014-09-22 16:43:00,354 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-8) [69585cfd] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDS >2014-09-22 16:43:00,366 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [69585cfd] Correlation ID: 69585cfd, Job ID: 1e3a6c24-4022-4d78-a71d-530106cdcbc3, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-09-22 16:43:01,504 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 9714e5d7-b19b-4335-9d96-d31983c4a471 : aqua-vds3.qa.lab.tlv.redhat.com >2014-09-22 16:43:01,528 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Available swap memory of host aqua-vds3.qa.lab.tlv.redhat.com [1023 MB] is under defined threshold [1024 MB]. >2014-09-22 16:43:01,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-13) Clearing cache of pool: 43e1738a-3a93-481a-befe-65c12fbc155f for problematic entities of VDS: aqua-vds3.qa.lab.tlv.redhat.com. >2014-09-22 16:43:01,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-13) Removing vds [9714e5d7-b19b-4335-9d96-d31983c4a471] from the domain in maintenance cache >2014-09-22 16:43:01,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-13) Removing host(s) [9714e5d7-b19b-4335-9d96-d31983c4a471] from hosts unseen domain report cache >2014-09-22 16:43:01,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) START, DisconnectStoragePoolVDSCommand(HostName = aqua-vds3.qa.lab.tlv.redhat.com, HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, vds_spm_id = 1), log id: 29bdbad1 >2014-09-22 16:43:04,729 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-2) [6bd81f04] Lock Acquired to object EngineLock [exclusiveLocks= key: 9714e5d7-b19b-4335-9d96-d31983c4a471 value: VDS >, sharedLocks= ] >2014-09-22 16:43:04,754 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 9714e5d7-b19b-4335-9d96-d31983c4a471 Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-22 16:43:04,872 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] START, RemoveVdsVDSCommand( HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471), log id: 4ca30620 >2014-09-22 16:43:06,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, DisconnectStoragePoolVDSCommand, log id: 29bdbad1 >2014-09-22 16:43:06,846 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-22 16:43:06,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [3cb372b3] START, DisconnectStorageServerVDSCommand( HostId = 9714e5d7-b19b-4335-9d96-d31983c4a471, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 213760dc >2014-09-22 16:43:07,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [3cb372b3] FINISH, DisconnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 213760dc >2014-09-22 16:43:07,104 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] vdsManager::disposing >2014-09-22 16:43:07,104 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] FINISH, RemoveVdsVDSCommand, log id: 4ca30620 >2014-09-22 16:43:07,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] Correlation ID: 6bd81f04, Call Stack: null, Custom Event ID: -1, Message: Host aqua-vds3.qa.lab.tlv.redhat.com was removed by admin. >2014-09-22 16:43:07,111 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-19) [6bd81f04] Lock freed to object EngineLock [exclusiveLocks= key: 9714e5d7-b19b-4335-9d96-d31983c4a471 value: VDS >, sharedLocks= ] >2014-09-22 17:42:44,706 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-09-22 17:42:47,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-09-22 18:42:44,747 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-98) Principal admin::internal synchronized >2014-09-22 18:42:47,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-22 19:42:44,760 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-30) Principal admin::internal synchronized >2014-09-22 19:42:47,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-09-22 20:42:44,777 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-09-22 20:42:47,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) [606158b9] Attempting to update VMs/Templates Ovf. >2014-09-22 21:42:44,808 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-09-22 21:42:47,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [632768a2] Attempting to update VMs/Templates Ovf. >2014-09-22 22:42:44,844 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-09-22 22:42:47,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-09-22 23:42:44,877 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized >2014-09-22 23:42:48,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-23 00:42:44,886 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-09-23 00:42:48,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-09-23 01:42:44,895 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-09-23 01:42:48,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-23 02:42:44,905 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-09-23 02:42:48,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-09-23 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) Start deleteAgedOutAuditLogs >2014-09-23 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-30) Start deleteAgedOutCommandEntities >2014-09-23 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) Finished deleteAgedOutAuditLogs >2014-09-23 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-30) Finished deleteAgedOutCommandEntities >2014-09-23 03:42:44,914 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-09-23 03:42:48,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-09-23 04:42:44,923 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [14e59306] Principal admin::internal synchronized >2014-09-23 04:42:48,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-23 05:42:44,933 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-09-23 05:42:48,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-09-23 06:42:44,943 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-09-23 06:42:48,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-09-23 07:42:44,953 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-09-23 07:42:48,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-09-23 08:42:44,961 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-09-23 08:42:48,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [164567a] Attempting to update VMs/Templates Ovf. >2014-09-23 09:42:44,971 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-09-23 09:42:48,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-09-23 10:42:44,983 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-09-23 10:42:48,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [14e59306] Attempting to update VMs/Templates Ovf. >2014-09-23 11:42:44,992 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-09-23 11:42:48,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-09-23 12:13:23,197 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-09-23 12:13:23,197 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-09-23 12:42:45,000 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-09-23 12:42:48,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-09-23 13:42:45,008 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-09-23 13:42:48,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [78d7871d] Attempting to update VMs/Templates Ovf. >2014-09-23 14:42:45,016 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-09-23 14:42:48,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Attempting to update VMs/Templates Ovf. >2014-09-23 15:42:45,023 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-23 15:42:48,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-09-23 16:42:45,032 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-09-23 16:42:48,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-09-23 17:42:45,042 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-09-23 17:42:48,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) [76aaf268] Attempting to update VMs/Templates Ovf. >2014-09-23 18:42:45,051 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-23 18:42:48,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-09-23 19:42:45,061 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-09-23 19:42:48,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) [62ac272] Attempting to update VMs/Templates Ovf. >2014-09-23 20:42:45,070 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-23 20:42:48,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-09-23 21:42:45,078 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-09-23 21:42:48,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) [325ae059] Attempting to update VMs/Templates Ovf. >2014-09-23 22:42:45,088 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) [76aaf268] Principal admin::internal synchronized >2014-09-23 22:42:48,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-09-23 23:42:45,096 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-09-23 23:42:48,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-09-24 00:42:45,105 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) [184f5e9a] Principal admin::internal synchronized >2014-09-24 00:42:48,526 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-09-24 01:42:45,116 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-24 01:42:48,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-09-24 02:42:45,124 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-09-24 02:42:48,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-09-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Start deleteAgedOutAuditLogs >2014-09-24 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-10) Start deleteAgedOutCommandEntities >2014-09-24 03:35:35,012 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Finished deleteAgedOutAuditLogs >2014-09-24 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-10) Finished deleteAgedOutCommandEntities >2014-09-24 03:42:45,133 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-09-24 03:42:48,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-24 04:42:45,142 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-24 04:42:48,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-09-24 05:42:45,153 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-09-24 05:42:48,631 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-09-24 06:42:45,162 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-09-24 06:42:48,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Attempting to update VMs/Templates Ovf. >2014-09-24 07:42:45,169 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-24 07:42:48,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-09-24 08:42:45,178 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-09-24 08:42:48,687 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-09-24 09:42:45,186 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-09-24 09:42:48,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-09-24 10:42:45,194 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [14e59306] Principal admin::internal synchronized >2014-09-24 10:42:48,724 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-09-24 10:51:25,803 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-24 10:51:25,803 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-24 10:51:46,109 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginAdminUserCommand internal: false. >2014-09-24 10:51:46,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 10:51:46,789 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-09-24 10:52:20,053 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-6) [5a9d323a] Lock Acquired to object EngineLock [exclusiveLocks= key: ea02ab0d-11dc-436f-a1ae-563a4c02d3b7 value: VM >, sharedLocks= ] >2014-09-24 10:52:20,335 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-6-thread-30) [5a9d323a] Running command: RemoveVmCommand internal: false. Entities affected : ID: ea02ab0d-11dc-436f-a1ae-563a4c02d3b7 Type: VMAction group DELETE_VM with role type USER >2014-09-24 10:52:20,341 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [5a9d323a] START, SetVmStatusVDSCommand( vmId = ea02ab0d-11dc-436f-a1ae-563a4c02d3b7, status = ImageLocked, exit status = Normal), log id: 56a42330 >2014-09-24 10:52:20,389 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-30) [5a9d323a] FINISH, SetVmStatusVDSCommand, log id: 56a42330 >2014-09-24 10:52:20,394 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-6-thread-30) [5a9d323a] Lock freed to object EngineLock [exclusiveLocks= key: ea02ab0d-11dc-436f-a1ae-563a4c02d3b7 value: VM >, sharedLocks= ] >2014-09-24 10:52:20,599 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-6-thread-30) [79bcf428] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: ea02ab0d-11dc-436f-a1ae-563a4c02d3b7 Type: VM >2014-09-24 10:52:20,654 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-6-thread-30) [7b975531] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-09-24 10:52:21,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-6-thread-30) [7b975531] START, DeleteImageGroupVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, storageDomainId = af66b5d4-dd51-491d-b2cb-c88c1ae89d41, imageGroupId = 7e06f7e1-da78-4e6a-967e-628d209c9cef, postZeros = false, forceDelete = false), log id: 405864b7 >2014-09-24 10:52:21,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-6-thread-30) [7b975531] FINISH, DeleteImageGroupVDSCommand, log id: 405864b7 >2014-09-24 10:52:21,444 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-30) [7b975531] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 968cdb4d-36b2-4c09-a6ce-73f92606736c >2014-09-24 10:52:21,444 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-6-thread-30) [7b975531] CommandMultiAsyncTasks::AttachTask: Attaching task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 to command 968cdb4d-36b2-4c09-a6ce-73f92606736c. >2014-09-24 10:52:21,522 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-30) [7b975531] Adding task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-09-24 10:52:21,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [7b975531] Correlation ID: 5a9d323a, Job ID: 3871593c-28b7-4ed8-b4be-91528bb293b4, Call Stack: null, Custom Event ID: -1, Message: VM shuki was successfully removed. >2014-09-24 10:52:21,947 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-30) [7b975531] BaseAsyncTask::startPollingTask: Starting to poll task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401. >2014-09-24 10:52:22,089 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-09-24 10:52:22,118 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) SPMAsyncTask::PollTask: Polling task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-09-24 10:52:22,127 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) BaseAsyncTask::onTaskEndSuccess: Task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-09-24 10:52:22,127 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) CommandAsyncTask::endActionIfNecessary: All tasks of command 968cdb4d-36b2-4c09-a6ce-73f92606736c has ended -> executing endAction >2014-09-24 10:52:22,127 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 968cdb4d-36b2-4c09-a6ce-73f92606736c): calling endAction . >2014-09-24 10:52:22,128 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-09-24 10:52:22,212 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-6-thread-45) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-09-24 10:52:22,214 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-09-24 10:52:22,214 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-09-24 10:52:22,220 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-45) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 >2014-09-24 10:52:22,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) START, SPMClearTaskVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, taskId = 17d5bf82-ea4e-4d64-9ce9-66fc65a30401), log id: 6353a8bb >2014-09-24 10:52:22,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) START, HSMClearTaskVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, taskId=17d5bf82-ea4e-4d64-9ce9-66fc65a30401), log id: 96abfef >2014-09-24 10:52:22,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) FINISH, HSMClearTaskVDSCommand, log id: 96abfef >2014-09-24 10:52:22,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) FINISH, SPMClearTaskVDSCommand, log id: 6353a8bb >2014-09-24 10:52:22,297 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-45) BaseAsyncTask::removeTaskFromDB: Removed task 17d5bf82-ea4e-4d64-9ce9-66fc65a30401 from DataBase >2014-09-24 10:52:22,298 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 968cdb4d-36b2-4c09-a6ce-73f92606736c >2014-09-24 10:53:22,364 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-9) [14e59306] Setting new tasks map. The map contains now 0 tasks >2014-09-24 10:53:22,364 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-9) [14e59306] Cleared all tasks of pool 43e1738a-3a93-481a-befe-65c12fbc155f. >2014-09-24 11:09:41,960 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-09-24 11:09:41,990 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:09:42,022 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [2b105286] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:09:42,080 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [3b090527] Running command: LogoutUserCommand internal: false. >2014-09-24 11:09:42,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [3b090527] Correlation ID: 3b090527, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:09:42,166 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: 2b105286, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:10:29,024 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-09-24 11:10:29,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:10:29,040 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [7e8fd1ed] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:10:29,046 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [3d5fd2aa] Running command: LogoutUserCommand internal: false. >2014-09-24 11:10:29,065 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3d5fd2aa] Correlation ID: 3d5fd2aa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:10:29,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: 7e8fd1ed, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:11:26,636 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-09-24 11:11:26,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:11:26,650 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [77623373] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:11:26,654 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [6c41fc81] Running command: LogoutUserCommand internal: false. >2014-09-24 11:11:26,670 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [6c41fc81] Correlation ID: 6c41fc81, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:11:26,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: 77623373, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:12:11,389 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-09-24 11:12:11,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:12:11,435 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [5d9535ac] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:12:11,440 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [526dbfe7] Running command: LogoutUserCommand internal: false. >2014-09-24 11:12:11,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [526dbfe7] Correlation ID: 526dbfe7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:12:11,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: 5d9535ac, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:12:16,792 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2014-09-24 11:12:16,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:13:07,925 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-09-24 11:13:07,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:13:07,940 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [1c56d20e] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:13:07,945 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [1604cad] Running command: LogoutUserCommand internal: false. >2014-09-24 11:13:07,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [1604cad] Correlation ID: 1604cad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:13:07,964 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: 1c56d20e, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:13:18,380 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2014-09-24 11:13:18,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:13:19,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:13:19,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:13:19,904 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-6) [hosts_update_6731d249-3216-42ff] Running command: UpdateVdsCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-24 11:13:19,932 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:13:19,933 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:13:20,482 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-6) Failed to run Power Management command on Host monique-vds05, no running proxy Host was found. >2014-09-24 11:13:20,529 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host monique-vds05.There is no other host in the data center that can be used to test the power management settings. >2014-09-24 11:13:20,572 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Kdump integration is enabled for host 'monique-vds05', but kdump is not configured properly on host. >2014-09-24 11:13:20,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: hosts_update_6731d249-3216-42ff, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 configuration was updated by admin. >2014-09-24 11:15:14,257 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-09-24 11:15:14,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:15:14,467 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [4abde436] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:15:14,474 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [2160a716] Running command: LogoutUserCommand internal: false. >2014-09-24 11:15:14,490 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [2160a716] Correlation ID: 2160a716, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:15:14,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: 4abde436, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:15:43,755 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:15:43,756 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:15:43,756 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-5) [hosts_update_62fff772-d2b5-4939] Running command: UpdateVdsCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-24 11:15:43,830 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:15:43,830 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 11:15:43,937 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-24 11:15:43,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: hosts_update_62fff772-d2b5-4939, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 configuration was updated by admin. >2014-09-24 11:15:51,201 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-09-24 11:15:51,242 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 11:15:51,276 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [1a7aad33] Running command: LogoutBySessionCommand internal: false. >2014-09-24 11:15:51,282 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [78d2537d] Running command: LogoutUserCommand internal: false. >2014-09-24 11:15:51,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [78d2537d] Correlation ID: 78d2537d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 11:15:51,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: 1a7aad33, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 11:42:45,211 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-09-24 11:42:48,746 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-09-24 11:42:48,879 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] Lock Acquired to object EngineLock [exclusiveLocks= key: bb435dee-edc6-4fb9-9d30-fb00c9fea902 value: DISK >key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1 value: DISK >key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 value: STORAGE >, sharedLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: OVF_UPDATE >] >2014-09-24 11:42:48,882 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 Type: Storage >2014-09-24 11:42:49,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] START, SetVolumeDescriptionVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, storageDomainId = af66b5d4-dd51-491d-b2cb-c88c1ae89d41, imageGroupId = af259134-1c2d-40e7-b6f7-44aa7ac7cef1, imageId = a557df0d-6c2f-4609-9f97-2e5eeaab4600), log id: 77b43066 >2014-09-24 11:42:49,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-09-24 11:42:49,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] -- setVolumeDescription parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGroupGUID=af259134-1c2d-40e7-b6f7-44aa7ac7cef1 > volUUID=a557df0d-6c2f-4609-9f97-2e5eeaab4600 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"af66b5d4-dd51-491d-b2cb-c88c1ae89d41"}],"Last Updated":"Wed Sep 10 14:36:07 IDT 2014"} > >2014-09-24 11:42:49,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [41e586b1] FINISH, SetVolumeDescriptionVDSCommand, log id: 77b43066 >2014-09-24 11:42:49,226 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] Running command: UploadStreamCommand internal: true. Entities affected : ID: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 Type: Storage >2014-09-24 11:42:49,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] START, UploadStreamVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 7a6df241 >2014-09-24 11:42:49,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- executeVdsBrokerCommand: >2014-09-24 11:42:49,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGUID=af259134-1c2d-40e7-b6f7-44aa7ac7cef1 > volUUID=a557df0d-6c2f-4609-9f97-2e5eeaab4600 > size=10,240 > >2014-09-24 11:42:49,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] FINISH, UploadStreamVDSCommand, log id: 7a6df241 >2014-09-24 11:42:49,474 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-10) [6abde349] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command db7bfd01-153a-4318-a2b8-81f305027eae >2014-09-24 11:42:49,474 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [6abde349] CommandMultiAsyncTasks::AttachTask: Attaching task 544104f7-0a0a-4d22-ab44-97a36e03a768 to command db7bfd01-153a-4318-a2b8-81f305027eae. >2014-09-24 11:42:49,557 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [6abde349] Adding task 544104f7-0a0a-4d22-ab44-97a36e03a768 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-09-24 11:42:49,576 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] Lock freed to object EngineLock [exclusiveLocks= key: bb435dee-edc6-4fb9-9d30-fb00c9fea902 value: DISK >key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1 value: DISK >key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 value: STORAGE >, sharedLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: OVF_UPDATE >] >2014-09-24 11:42:49,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] START, SetVolumeDescriptionVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, storageDomainId = af66b5d4-dd51-491d-b2cb-c88c1ae89d41, imageGroupId = af259134-1c2d-40e7-b6f7-44aa7ac7cef1, imageId = a557df0d-6c2f-4609-9f97-2e5eeaab4600), log id: 53e1a144 >2014-09-24 11:42:49,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-09-24 11:42:49,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- setVolumeDescription parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGroupGUID=af259134-1c2d-40e7-b6f7-44aa7ac7cef1 > volUUID=a557df0d-6c2f-4609-9f97-2e5eeaab4600 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"af66b5d4-dd51-491d-b2cb-c88c1ae89d41"}],"Last Updated":"Wed Sep 24 11:42:48 IDT 2014","Size":10240} > >2014-09-24 11:42:49,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] FINISH, SetVolumeDescriptionVDSCommand, log id: 53e1a144 >2014-09-24 11:42:49,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] START, SetVolumeDescriptionVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, storageDomainId = af66b5d4-dd51-491d-b2cb-c88c1ae89d41, imageGroupId = bb435dee-edc6-4fb9-9d30-fb00c9fea902, imageId = e1bc8af1-5f34-4d99-b065-80dcf966d308), log id: 6a208741 >2014-09-24 11:42:49,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-09-24 11:42:49,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] -- setVolumeDescription parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGroupGUID=bb435dee-edc6-4fb9-9d30-fb00c9fea902 > volUUID=e1bc8af1-5f34-4d99-b065-80dcf966d308 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"af66b5d4-dd51-491d-b2cb-c88c1ae89d41"}],"Last Updated":"Wed Sep 10 14:36:07 IDT 2014"} > >2014-09-24 11:42:49,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [6abde349] FINISH, SetVolumeDescriptionVDSCommand, log id: 6a208741 >2014-09-24 11:42:50,029 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Running command: UploadStreamCommand internal: true. Entities affected : ID: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 Type: Storage >2014-09-24 11:42:50,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] START, UploadStreamVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 4a061cc3 >2014-09-24 11:42:50,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] -- executeVdsBrokerCommand: >2014-09-24 11:42:50,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] -- parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGUID=bb435dee-edc6-4fb9-9d30-fb00c9fea902 > volUUID=e1bc8af1-5f34-4d99-b065-80dcf966d308 > size=10,240 > >2014-09-24 11:42:50,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] FINISH, UploadStreamVDSCommand, log id: 4a061cc3 >2014-09-24 11:42:50,283 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [362bc3f3] CommandMultiAsyncTasks::AttachTask: Attaching task cdab9404-b9fa-4d50-baa5-13923f0e1b59 to command db7bfd01-153a-4318-a2b8-81f305027eae. >2014-09-24 11:42:50,382 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Adding task cdab9404-b9fa-4d50-baa5-13923f0e1b59 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-09-24 11:42:50,444 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Trying to release a shared lock for key: 43e1738a-3a93-481a-befe-65c12fbc155fOVF_UPDATE , but lock does not exist >2014-09-24 11:42:50,444 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Trying to release exclusive lock which does not exist, lock key: bb435dee-edc6-4fb9-9d30-fb00c9fea902DISK >2014-09-24 11:42:50,444 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Trying to release exclusive lock which does not exist, lock key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1DISK >2014-09-24 11:42:50,444 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Trying to release exclusive lock which does not exist, lock key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41STORAGE >2014-09-24 11:42:50,444 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Lock freed to object EngineLock [exclusiveLocks= key: bb435dee-edc6-4fb9-9d30-fb00c9fea902 value: DISK >key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1 value: DISK >key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 value: STORAGE >, sharedLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: OVF_UPDATE >] >2014-09-24 11:42:50,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] START, SetVolumeDescriptionVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, storageDomainId = af66b5d4-dd51-491d-b2cb-c88c1ae89d41, imageGroupId = bb435dee-edc6-4fb9-9d30-fb00c9fea902, imageId = e1bc8af1-5f34-4d99-b065-80dcf966d308), log id: 3f541217 >2014-09-24 11:42:50,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-09-24 11:42:50,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] -- setVolumeDescription parameters: > spUUID=43e1738a-3a93-481a-befe-65c12fbc155f > sdUUID=af66b5d4-dd51-491d-b2cb-c88c1ae89d41 > imageGroupGUID=bb435dee-edc6-4fb9-9d30-fb00c9fea902 > volUUID=e1bc8af1-5f34-4d99-b065-80dcf966d308 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"af66b5d4-dd51-491d-b2cb-c88c1ae89d41"}],"Last Updated":"Wed Sep 24 11:42:48 IDT 2014","Size":10240} > >2014-09-24 11:42:50,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-10) [362bc3f3] FINISH, SetVolumeDescriptionVDSCommand, log id: 3f541217 >2014-09-24 11:42:50,793 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [362bc3f3] BaseAsyncTask::startPollingTask: Starting to poll task 544104f7-0a0a-4d22-ab44-97a36e03a768. >2014-09-24 11:42:50,793 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [362bc3f3] BaseAsyncTask::startPollingTask: Starting to poll task cdab9404-b9fa-4d50-baa5-13923f0e1b59. >2014-09-24 11:42:52,480 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-72) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-72) SPMAsyncTask::PollTask: Polling task cdab9404-b9fa-4d50-baa5-13923f0e1b59 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-72) BaseAsyncTask::onTaskEndSuccess: Task cdab9404-b9fa-4d50-baa5-13923f0e1b59 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-72) Task with DB Task ID 152e19fd-c3c4-4758-b207-f1cc03023260 and VDSM Task ID 544104f7-0a0a-4d22-ab44-97a36e03a768 is in state Polling. End action for command db7bfd01-153a-4318-a2b8-81f305027eae will proceed when all the entitys tasks are completed. >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-72) SPMAsyncTask::PollTask: Polling task 544104f7-0a0a-4d22-ab44-97a36e03a768 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-72) BaseAsyncTask::onTaskEndSuccess: Task 544104f7-0a0a-4d22-ab44-97a36e03a768 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-09-24 11:42:52,521 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-72) CommandAsyncTask::endActionIfNecessary: All tasks of command db7bfd01-153a-4318-a2b8-81f305027eae has ended -> executing endAction >2014-09-24 11:42:52,522 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-72) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: db7bfd01-153a-4318-a2b8-81f305027eae): calling endAction . >2014-09-24 11:42:52,522 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-36) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-09-24 11:42:52,568 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-09-24 11:42:52,569 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Trying to release a shared lock for key: 43e1738a-3a93-481a-befe-65c12fbc155fOVF_UPDATE , but lock does not exist >2014-09-24 11:42:52,569 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Trying to release exclusive lock which does not exist, lock key: bb435dee-edc6-4fb9-9d30-fb00c9fea902DISK >2014-09-24 11:42:52,569 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Trying to release exclusive lock which does not exist, lock key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1DISK >2014-09-24 11:42:52,569 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Trying to release exclusive lock which does not exist, lock key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41STORAGE >2014-09-24 11:42:52,569 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] Lock freed to object EngineLock [exclusiveLocks= key: bb435dee-edc6-4fb9-9d30-fb00c9fea902 value: DISK >key: af259134-1c2d-40e7-b6f7-44aa7ac7cef1 value: DISK >key: af66b5d4-dd51-491d-b2cb-c88c1ae89d41 value: STORAGE >, sharedLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: OVF_UPDATE >] >2014-09-24 11:42:52,569 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-09-24 11:42:52,569 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-09-24 11:42:52,569 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] SPMAsyncTask::ClearAsyncTask: Attempting to clear task cdab9404-b9fa-4d50-baa5-13923f0e1b59 >2014-09-24 11:42:52,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] START, SPMClearTaskVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, taskId = cdab9404-b9fa-4d50-baa5-13923f0e1b59), log id: 33f30eff >2014-09-24 11:42:52,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] START, HSMClearTaskVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, taskId=cdab9404-b9fa-4d50-baa5-13923f0e1b59), log id: 1a5da1fe >2014-09-24 11:42:52,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] FINISH, HSMClearTaskVDSCommand, log id: 1a5da1fe >2014-09-24 11:42:52,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] FINISH, SPMClearTaskVDSCommand, log id: 33f30eff >2014-09-24 11:42:52,660 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] BaseAsyncTask::removeTaskFromDB: Removed task cdab9404-b9fa-4d50-baa5-13923f0e1b59 from DataBase >2014-09-24 11:42:52,660 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 544104f7-0a0a-4d22-ab44-97a36e03a768 >2014-09-24 11:42:52,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] START, SPMClearTaskVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false, taskId = 544104f7-0a0a-4d22-ab44-97a36e03a768), log id: 2691be72 >2014-09-24 11:42:52,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] START, HSMClearTaskVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, taskId=544104f7-0a0a-4d22-ab44-97a36e03a768), log id: 2a6e4226 >2014-09-24 11:42:52,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] FINISH, HSMClearTaskVDSCommand, log id: 2a6e4226 >2014-09-24 11:42:52,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-36) [41e586b1] FINISH, SPMClearTaskVDSCommand, log id: 2691be72 >2014-09-24 11:42:52,749 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] BaseAsyncTask::removeTaskFromDB: Removed task 544104f7-0a0a-4d22-ab44-97a36e03a768 from DataBase >2014-09-24 11:42:52,749 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-36) [41e586b1] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity db7bfd01-153a-4318-a2b8-81f305027eae >2014-09-24 11:44:22,477 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) Setting new tasks map. The map contains now 0 tasks >2014-09-24 11:44:22,477 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) Cleared all tasks of pool 43e1738a-3a93-481a-befe-65c12fbc155f. >2014-09-24 12:42:45,217 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-09-24 12:42:50,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-09-24 13:07:50,931 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-09-24 13:07:50,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:07:56,705 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [6cb85da4] Running command: LogoutUserCommand internal: false. >2014-09-24 13:07:56,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [6cb85da4] Correlation ID: 6cb85da4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:07:56,732 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [759e0ed1] Running command: LogoutUserCommand internal: false. >2014-09-24 13:07:56,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [759e0ed1] Correlation ID: 759e0ed1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:07:56,790 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [c8604ad] Running command: LogoutUserCommand internal: false. >2014-09-24 13:07:56,802 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [b050279] Running command: LogoutUserCommand internal: false. >2014-09-24 13:07:56,809 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [435c697b] Running command: LogoutUserCommand internal: false. >2014-09-24 13:07:56,867 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [c8604ad] Correlation ID: c8604ad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:07:56,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [435c697b] Correlation ID: 435c697b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:07:56,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [b050279] Correlation ID: b050279, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:07:57,102 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-24 13:07:57,102 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-24 13:17:50,706 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-09-24 13:17:50,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:17:50,733 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [72b36c4d] Running command: LogoutBySessionCommand internal: false. >2014-09-24 13:17:50,748 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [3e5c7b0e] Running command: LogoutUserCommand internal: false. >2014-09-24 13:17:50,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3e5c7b0e] Correlation ID: 3e5c7b0e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:17:50,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: 72b36c4d, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 13:18:36,403 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-09-24 13:18:36,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:18:36,558 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [3030eefa] Running command: LogoutBySessionCommand internal: false. >2014-09-24 13:18:36,582 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [c1e7ed3] Running command: LogoutUserCommand internal: false. >2014-09-24 13:18:36,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [c1e7ed3] Correlation ID: c1e7ed3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:18:36,626 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: 3030eefa, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 13:18:42,066 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-09-24 13:18:42,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:18:43,449 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:18:43,449 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:18:43,450 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-8) [hosts_update_f0872bf8-8582-4f80] Running command: UpdateVdsCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-24 13:18:43,495 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:18:43,496 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:18:44,336 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-8) Failed to run Power Management command on Host monique-vds05, no running proxy Host was found. >2014-09-24 13:18:44,409 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host monique-vds05.There is no other host in the data center that can be used to test the power management settings. >2014-09-24 13:18:44,483 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Kdump integration is enabled for host 'monique-vds05', but kdump is not configured properly on host. >2014-09-24 13:18:44,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: hosts_update_f0872bf8-8582-4f80, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 configuration was updated by admin. >2014-09-24 13:19:13,150 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-09-24 13:19:13,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:19:13,295 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [5d42773d] Running command: LogoutBySessionCommand internal: false. >2014-09-24 13:19:13,299 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [54cc9bba] Running command: LogoutUserCommand internal: false. >2014-09-24 13:19:13,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [54cc9bba] Correlation ID: 54cc9bba, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:19:13,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: 5d42773d, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 13:20:05,959 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:20:05,960 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:20:05,960 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-12) [hosts_update_2d447323-cd37-427b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-24 13:20:05,975 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:20:05,975 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-24 13:20:06,049 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-24 13:20:06,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-12) Correlation ID: hosts_update_2d447323-cd37-427b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 configuration was updated by admin. >2014-09-24 13:20:17,416 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-09-24 13:20:17,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-24 13:20:17,454 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [13b49045] Running command: LogoutBySessionCommand internal: false. >2014-09-24 13:20:17,460 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [b5b3ab9] Running command: LogoutUserCommand internal: false. >2014-09-24 13:20:17,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [b5b3ab9] Correlation ID: b5b3ab9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-24 13:20:17,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: 13b49045, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-24 13:42:45,225 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-09-24 13:42:50,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-24 14:42:45,235 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-09-24 14:42:50,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-09-24 15:42:45,246 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) Principal admin::internal synchronized >2014-09-24 15:42:50,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-09-24 16:42:45,255 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-09-24 16:42:50,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-09-24 17:42:45,263 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-09-24 17:42:50,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-09-24 18:42:45,271 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-24 18:42:50,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-09-24 19:42:45,283 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-09-24 19:42:50,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-09-24 20:42:45,302 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-41) [62ac272] Principal admin::internal synchronized >2014-09-24 20:42:50,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-09-24 21:42:45,310 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Principal admin::internal synchronized >2014-09-24 21:42:50,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) [606158b9] Attempting to update VMs/Templates Ovf. >2014-09-24 22:42:45,318 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-09-24 22:42:50,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-09-24 23:42:45,327 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) [37ae315a] Principal admin::internal synchronized >2014-09-24 23:42:50,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-09-25 00:42:45,336 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-09-25 00:42:51,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-09-25 01:42:45,343 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-09-25 01:42:51,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-09-25 02:42:45,358 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) [184f5e9a] Principal admin::internal synchronized >2014-09-25 02:42:51,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-09-25 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-76) Start deleteAgedOutAuditLogs >2014-09-25 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-5) Start deleteAgedOutCommandEntities >2014-09-25 03:35:35,013 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-5) Finished deleteAgedOutCommandEntities >2014-09-25 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-76) Finished deleteAgedOutAuditLogs >2014-09-25 03:42:45,381 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized >2014-09-25 03:42:51,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [184f5e9a] Attempting to update VMs/Templates Ovf. >2014-09-25 04:42:45,390 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Principal admin::internal synchronized >2014-09-25 04:42:51,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-09-25 05:42:45,400 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-25 05:42:51,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [37ae315a] Attempting to update VMs/Templates Ovf. >2014-09-25 06:42:45,408 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-25 06:42:51,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-09-25 07:42:45,419 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-09-25 07:42:51,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-09-25 08:42:45,428 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [333b6810] Principal admin::internal synchronized >2014-09-25 08:42:51,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-09-25 09:42:45,441 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) [37ae315a] Principal admin::internal synchronized >2014-09-25 09:42:51,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) [333b6810] Attempting to update VMs/Templates Ovf. >2014-09-25 10:42:45,451 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-25 10:42:51,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-09-25 11:42:45,459 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-09-25 11:42:51,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-09-25 12:42:45,468 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-25 12:42:51,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-25 13:42:45,477 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Principal admin::internal synchronized >2014-09-25 13:42:51,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-09-25 14:42:45,486 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-09-25 14:42:51,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Attempting to update VMs/Templates Ovf. >2014-09-25 15:42:45,494 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-09-25 15:42:51,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-09-25 16:42:45,504 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-25 16:42:51,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-09-25 17:42:45,514 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-25 17:42:51,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-25 18:42:45,524 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [632768a2] Principal admin::internal synchronized >2014-09-25 18:42:51,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-09-25 19:42:45,531 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-31) [325ae059] Principal admin::internal synchronized >2014-09-25 19:42:51,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-09-25 20:42:45,540 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) [735ee4d8] Principal admin::internal synchronized >2014-09-25 20:42:51,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-09-25 21:42:45,550 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-09-25 21:42:51,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-09-25 22:42:45,565 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-88) Principal admin::internal synchronized >2014-09-25 22:42:51,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-09-25 23:42:45,572 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-09-25 23:42:51,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-09-26 00:42:45,582 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-09-26 00:42:51,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-09-26 01:42:45,593 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-09-26 01:42:51,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [37ae315a] Attempting to update VMs/Templates Ovf. >2014-09-26 02:42:45,601 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-09-26 02:42:51,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-09-26 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-41) [62ac272] Start deleteAgedOutAuditLogs >2014-09-26 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Start deleteAgedOutCommandEntities >2014-09-26 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-41) [62ac272] Finished deleteAgedOutAuditLogs >2014-09-26 03:35:35,013 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Finished deleteAgedOutCommandEntities >2014-09-26 03:42:45,609 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized >2014-09-26 03:42:51,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-09-26 04:42:45,617 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) Principal admin::internal synchronized >2014-09-26 04:42:51,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [19e1cd7e] Attempting to update VMs/Templates Ovf. >2014-09-26 05:42:45,650 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized >2014-09-26 05:42:51,593 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-26 06:42:45,661 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-09-26 06:42:51,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-09-26 07:42:45,668 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-56) Principal admin::internal synchronized >2014-09-26 07:42:51,629 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-09-26 08:42:45,677 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized >2014-09-26 08:42:51,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-09-26 09:42:45,684 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-26 09:42:51,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-09-26 10:42:45,693 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-09-26 10:42:51,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-09-26 11:42:45,704 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) [606158b9] Principal admin::internal synchronized >2014-09-26 11:42:51,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-09-26 12:42:45,713 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-09-26 12:42:51,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-09-26 13:42:45,722 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-56) Principal admin::internal synchronized >2014-09-26 13:42:51,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-09-26 14:42:45,736 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-09-26 14:42:51,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Attempting to update VMs/Templates Ovf. >2014-09-26 15:42:45,748 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [632768a2] Principal admin::internal synchronized >2014-09-26 15:42:51,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-09-26 16:42:45,758 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-09-26 16:42:51,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-09-26 17:42:45,767 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-09-26 17:42:51,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-09-26 18:42:45,775 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-09-26 18:42:51,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-09-26 19:42:45,783 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-09-26 19:42:51,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) [606158b9] Attempting to update VMs/Templates Ovf. >2014-09-26 20:42:45,791 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-09-26 20:42:51,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-09-26 21:42:45,806 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) [6293e326] Principal admin::internal synchronized >2014-09-26 21:42:51,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-09-26 22:42:45,815 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-28) Principal admin::internal synchronized >2014-09-26 22:42:51,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-09-26 23:42:45,823 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-09-26 23:42:51,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-27 00:42:45,831 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-27 00:42:51,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-09-27 01:42:45,839 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-09-27 01:42:51,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-09-27 02:42:45,848 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-32) Principal admin::internal synchronized >2014-09-27 02:42:52,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-09-27 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-80) Start deleteAgedOutAuditLogs >2014-09-27 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) [325ae059] Start deleteAgedOutCommandEntities >2014-09-27 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-80) Finished deleteAgedOutAuditLogs >2014-09-27 03:35:35,012 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) [325ae059] Finished deleteAgedOutCommandEntities >2014-09-27 03:42:45,857 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-09-27 03:42:52,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-09-27 04:42:45,872 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-27 04:42:52,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Attempting to update VMs/Templates Ovf. >2014-09-27 05:42:45,881 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-09-27 05:42:52,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) [aa66c2b] Attempting to update VMs/Templates Ovf. >2014-09-27 06:42:45,889 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [333b6810] Principal admin::internal synchronized >2014-09-27 06:42:52,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-09-27 07:42:45,915 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-27 07:42:52,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-09-27 08:42:45,946 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-09-27 08:42:52,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-09-27 09:42:45,975 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Principal admin::internal synchronized >2014-09-27 09:42:52,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-09-27 10:42:45,984 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-36) Principal admin::internal synchronized >2014-09-27 10:42:52,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-09-27 11:42:45,992 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-09-27 11:42:52,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-09-27 12:42:46,032 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) [606158b9] Principal admin::internal synchronized >2014-09-27 12:42:52,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-27 13:42:46,040 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-09-27 13:42:52,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) [76aaf268] Attempting to update VMs/Templates Ovf. >2014-09-27 14:42:46,049 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-09-27 14:42:52,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-09-27 15:42:46,057 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-09-27 15:42:52,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-09-27 16:42:46,065 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-09-27 16:42:52,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-09-27 17:42:46,075 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-09-27 17:42:52,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-09-27 18:42:46,083 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-27 18:42:52,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-09-27 19:42:46,091 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-09-27 19:42:52,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-09-27 20:42:46,099 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-27 20:42:52,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-09-27 21:42:46,118 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized >2014-09-27 21:42:52,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [362bc3f3] Attempting to update VMs/Templates Ovf. >2014-09-27 22:42:46,134 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-27 22:42:52,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) [4ab5d8c5] Attempting to update VMs/Templates Ovf. >2014-09-27 23:42:46,147 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-09-27 23:42:52,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-09-28 00:42:46,166 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-32) Principal admin::internal synchronized >2014-09-28 00:42:52,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-09-28 01:42:46,175 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-28 01:42:52,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-09-28 02:42:46,184 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-28 02:42:52,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) [6293e326] Attempting to update VMs/Templates Ovf. >2014-09-28 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-58) [13609e21] Start deleteAgedOutAuditLogs >2014-09-28 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-85) Start deleteAgedOutCommandEntities >2014-09-28 03:35:35,020 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-85) Finished deleteAgedOutCommandEntities >2014-09-28 03:35:35,021 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-58) [13609e21] Finished deleteAgedOutAuditLogs >2014-09-28 03:42:46,192 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-09-28 03:42:52,538 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-09-28 04:42:46,200 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-09-28 04:42:52,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-09-28 05:42:46,209 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-09-28 05:42:52,573 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) [76aaf268] Attempting to update VMs/Templates Ovf. >2014-09-28 06:42:46,217 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) [13609e21] Principal admin::internal synchronized >2014-09-28 06:42:52,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-09-28 07:42:46,236 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-09-28 07:42:52,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) [6293e326] Attempting to update VMs/Templates Ovf. >2014-09-28 08:42:46,251 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-09-28 08:42:52,629 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) [4ab5d8c5] Attempting to update VMs/Templates Ovf. >2014-09-28 09:42:46,262 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-09-28 09:42:52,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-09-28 10:42:46,271 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-09-28 10:42:52,721 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-09-28 11:42:46,279 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-09-28 11:42:52,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-09-28 12:42:46,286 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [14e59306] Principal admin::internal synchronized >2014-09-28 12:42:52,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) [3cb372b3] Attempting to update VMs/Templates Ovf. >2014-09-28 13:42:46,294 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized >2014-09-28 13:42:52,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-09-28 14:42:46,303 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) [76aaf268] Principal admin::internal synchronized >2014-09-28 14:42:52,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-09-28 15:42:46,312 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-09-28 15:42:52,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-09-28 16:42:46,320 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-09-28 16:42:52,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-09-28 17:42:46,330 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) [13609e21] Principal admin::internal synchronized >2014-09-28 17:42:52,853 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-09-28 18:42:46,340 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-09-28 18:42:52,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) [62ac272] Attempting to update VMs/Templates Ovf. >2014-09-28 19:42:46,348 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-09-28 19:42:52,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-09-28 20:42:46,356 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-09-28 20:42:52,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-09-28 21:42:46,365 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-09-28 21:42:52,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-09-28 22:42:46,374 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-09-28 22:42:52,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-09-28 23:42:46,392 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-09-28 23:42:52,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-09-29 00:42:46,401 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-09-29 00:42:52,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-09-29 01:42:46,409 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) [37ae315a] Principal admin::internal synchronized >2014-09-29 01:42:53,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-09-29 02:42:46,417 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-29 02:42:53,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-09-29 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Start deleteAgedOutCommandEntities >2014-09-29 03:35:35,005 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-4) Start deleteAgedOutAuditLogs >2014-09-29 03:35:35,016 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-4) Finished deleteAgedOutAuditLogs >2014-09-29 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Finished deleteAgedOutCommandEntities >2014-09-29 03:42:46,425 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-93) Principal admin::internal synchronized >2014-09-29 03:42:53,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-09-29 04:42:46,436 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-29 04:42:53,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-29 05:42:46,444 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) [13609e21] Principal admin::internal synchronized >2014-09-29 05:42:53,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-09-29 06:42:46,453 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) [606158b9] Principal admin::internal synchronized >2014-09-29 06:42:53,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [13609e21] Attempting to update VMs/Templates Ovf. >2014-09-29 07:42:46,461 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-09-29 07:42:53,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) [4ab5d8c5] Attempting to update VMs/Templates Ovf. >2014-09-29 08:42:46,470 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-09-29 08:42:53,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [164567a] Attempting to update VMs/Templates Ovf. >2014-09-29 09:42:46,478 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-09-29 09:42:53,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-09-29 10:01:18,346 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-09-29 10:01:18,346 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-09-29 10:04:04,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 10:07:21,862 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-29 10:07:21,898 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-29 10:07:21,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-29 10:07:21,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-29 10:07:21,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-29 10:07:21,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-29 10:07:21,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-29 10:07:21,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-29 10:07:21,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-29 10:07:21,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-29 10:07:21,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-29 10:07:21,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DB_USER" is "engine". >2014-09-29 10:07:21,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-29 10:07:21,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-29 10:07:21,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-29 10:07:21,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-29 10:07:21,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-29 10:07:21,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-29 10:07:21,909 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-29 10:07:21,909 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-29 10:07:21,909 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-29 10:07:21,952 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_USER" is "ovirt". >2014-09-29 10:07:21,952 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-29 10:07:21,952 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-29 10:07:21,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-29 10:07:21,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-29 10:07:21,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-29 10:07:21,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 40) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-29 10:07:39,977 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Start initializing Backend >2014-09-29 10:07:40,982 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 43) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-29 10:07:41,282 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Running ovirt-engine 3.5.0-0.13.beta.el6ev >2014-09-29 10:07:41,283 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 43) Start initializing dictionaries >2014-09-29 10:07:41,288 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 43) Finished initializing dictionaries >2014-09-29 10:07:41,288 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Start initializing AuditLogCleanupManager >2014-09-29 10:07:41,289 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-29 10:07:41,374 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Finished initializing AuditLogCleanupManager >2014-09-29 10:07:41,374 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Start initializing CommandEntityCleanupManager >2014-09-29 10:07:41,375 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-29 10:07:41,377 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Finished initializing CommandEntityCleanupManager >2014-09-29 10:07:41,399 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Start initializing TagsDirector >2014-09-29 10:07:41,418 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Tag root added to tree >2014-09-29 10:07:41,438 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Finished initializing TagsDirector >2014-09-29 10:07:41,438 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 43) Start initializing IsoDomainListSyncronizer >2014-09-29 10:07:41,449 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 43) Finished initializing IsoDomainListSyncronizer >2014-09-29 10:07:41,563 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Completed initializing handlers >2014-09-29 10:07:41,679 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Start initializing ErrorTranslatorImpl >2014-09-29 10:07:41,748 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-29 10:07:41,750 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Finished initializing ErrorTranslatorImpl >2014-09-29 10:07:41,750 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Start initializing ErrorTranslatorImpl >2014-09-29 10:07:41,751 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Finished initializing ErrorTranslatorImpl >2014-09-29 10:07:41,751 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Mark incomplete jobs as UNKNOWN >2014-09-29 10:07:41,980 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 43) Start initializing JobRepositoryCleanupManager >2014-09-29 10:07:41,982 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 43) Finished initializing JobRepositoryCleanupManager >2014-09-29 10:07:41,982 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 43) Start initializing AutoRecoveryManager >2014-09-29 10:07:41,984 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 43) Finished initializing AutoRecoveryManager >2014-09-29 10:07:41,987 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 43) Start initializing ExecutionMessageDirector >2014-09-29 10:07:41,989 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 43) Finished initializing ExecutionMessageDirector >2014-09-29 10:07:42,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Loading extension 'builtin-authn-internal' >2014-09-29 10:07:42,167 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'builtin-authn-internal' loaded >2014-09-29 10:07:42,170 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Loading extension 'internal' >2014-09-29 10:07:42,172 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'internal' loaded >2014-09-29 10:07:42,174 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Initializing extension 'builtin-authn-internal' >2014-09-29 10:07:42,174 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'builtin-authn-internal' initialized >2014-09-29 10:07:42,175 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Initializing extension 'internal' >2014-09-29 10:07:42,175 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'internal' initialized >2014-09-29 10:07:42,176 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Start of enabled extensions list >2014-09-29 10:07:42,191 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 10:07:42,192 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 10:07:42,192 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) End of enabled extensions list >2014-09-29 10:07:42,451 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 41) Start initializing DbUserCacheManager >2014-09-29 10:07:42,470 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 41) Finished initializing DbUserCacheManager >2014-09-29 10:07:42,510 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 41) Initialization of AsyncTaskManager completed successfully. >2014-09-29 10:07:42,512 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 41) Start initializing ResourceManager >2014-09-29 10:07:42,735 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 41) Entered VdsManager constructor >2014-09-29 10:07:42,858 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 41) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-09-29 10:07:43,354 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-09-29 10:07:43,402 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 41) VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 was added to the Resource Manager >2014-09-29 10:07:43,479 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 41) Finished initializing ResourceManager >2014-09-29 10:07:43,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 41) Initialization of OvfDataUpdater completed successfully. >2014-09-29 10:07:43,503 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Start scheduling to enable vds load balancer >2014-09-29 10:07:43,504 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Finished scheduling to enable vds load balancer >2014-09-29 10:07:43,504 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Start HA Reservation check >2014-09-29 10:07:43,505 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Finished HA Reservation check >2014-09-29 10:07:43,532 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 41) Init VM custom properties utilities >2014-09-29 10:07:43,533 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 41) Init device custom properties utilities >2014-09-29 10:07:43,538 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Initializing Scheduling manager >2014-09-29 10:07:43,565 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-09-29 10:07:43,639 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-09-29 10:07:43,669 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) External scheduler disabled, discovery skipped >2014-09-29 10:07:43,670 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 41) Initialized Scheduling manager >2014-09-29 10:07:43,670 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 41) Initializing DWH Heart Beat >2014-09-29 10:07:43,671 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 41) DWH Heart Beat initialized >2014-09-29 10:07:47,670 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-29 10:07:47,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-29 10:07:47,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-29 10:07:47,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-29 10:07:47,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-29 10:07:48,288 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 539 seconds while maximum configured value is 300 seconds. >2014-09-29 10:07:48,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 55ec63fe >2014-09-29 10:07:48,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 55ec63fe >2014-09-29 10:07:49,386 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds05 >2014-09-29 10:07:49,899 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [3d437469] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 10:07:53,632 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [5a652025] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 10:07:53,653 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [5a652025] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-29 10:07:53,814 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [5a652025] Correlation ID: 5a652025, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-09-29 10:07:54,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [5a652025] hostFromVds::selectedVds - monique-vds05, spmStatus SPM, storage pool DC1 >2014-09-29 10:07:54,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [5a652025] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-09-29 10:07:54,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [5a652025] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05 (Address: monique-vds05.tlv.redhat.com). >2014-09-29 10:07:54,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-12) [5a652025] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: 4f026ffa >2014-09-29 10:07:55,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-12) [5a652025] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-29 10:07:55,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-12) [5a652025] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 4d1b5cb0 >2014-09-29 10:07:55,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-12) [5a652025] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d1b5cb0 >2014-09-29 10:07:55,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-12) [5a652025] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f026ffa >2014-09-29 10:07:55,089 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-12) [5a652025] Discovered no tasks on Storage Pool DC1 >2014-09-29 11:07:43,394 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-09-29 11:07:43,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-09-29 11:07:59,535 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 540 seconds while maximum configured value is 300 seconds. >2014-09-29 11:29:30,132 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-3) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-29 11:29:30,204 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-3) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-29 11:29:30,250 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-29 11:29:30,251 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-29 11:29:43,015 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginAdminUserCommand internal: false. >2014-09-29 11:29:43,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-29 11:29:52,056 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-09-29 11:29:52,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-29 11:35:28,408 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-6) [7a9bdf86] Running command: AddVdsCommand internal: false. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-29 11:35:28,669 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [70de2203] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 11:35:28,720 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [70de2203] Lock freed to object EngineLock [exclusiveLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: REGISTER_VDS >, sharedLocks= ] >2014-09-29 11:35:28,791 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] START, RemoveVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 5e051799 >2014-09-29 11:35:28,791 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] FINISH, RemoveVdsVDSCommand, log id: 5e051799 >2014-09-29 11:35:28,861 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] START, AddVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 377ad69d >2014-09-29 11:35:28,862 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] AddVds - entered , starting logic to add VDS 08c6d6dd-82dd-41e7-9e92-6957151a40e3 >2014-09-29 11:35:28,921 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] AddVds - VDS 08c6d6dd-82dd-41e7-9e92-6957151a40e3 was added, will try to add it to the resource manager >2014-09-29 11:35:28,932 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [70de2203] Entered VdsManager constructor >2014-09-29 11:35:28,957 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [70de2203] Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-09-29 11:35:29,101 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [70de2203] VDS 08c6d6dd-82dd-41e7-9e92-6957151a40e3 was added to the Resource Manager >2014-09-29 11:35:29,102 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] FINISH, AddVdsVDSCommand, log id: 377ad69d >2014-09-29 11:35:29,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [70de2203] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 11:35:29,587 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-6) [70de2203] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 11:35:29,663 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-6) [70de2203] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 11:35:29,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 578d44be >2014-09-29 11:35:30,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [70de2203] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 578d44be >2014-09-29 11:35:30,542 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS >, sharedLocks= ] >2014-09-29 11:35:30,582 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 11:35:30,591 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] Before Installation host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 11:35:30,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [70de2203] Correlation ID: 7a9bdf86, Job ID: ead92c25-0512-4d82-a771-05f1ba85c8a3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was added by admin. >2014-09-29 11:35:30,709 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 67ac4184 >2014-09-29 11:35:30,715 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] FINISH, SetVdsStatusVDSCommand, log id: 67ac4184 >2014-09-29 11:35:30,854 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-41) [20b19173] Installation rose07.qa.lab.tlv.redhat.com: Connected to host rose07.qa.lab.tlv.redhat.com with SSH key fingerprint: ed:27:fb:75:a2:a0:82:18:3f:90:38:f5:44:3c:ee:61 >2014-09-29 11:35:31,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-41) [20b19173] Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Connected to host rose07.qa.lab.tlv.redhat.com with SSH key fingerprint: ed:27:fb:75:a2:a0:82:18:3f:90:38:f5:44:3c:ee:61. >2014-09-29 11:35:31,034 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-41) [20b19173] Installation of rose07.qa.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-29 11:35:31,071 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-7-thread-41) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is out of date, re-creating >2014-09-29 11:35:31,563 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-41) SSH execute root@rose07.qa.lab.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-29 11:35:32,356 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-09-29 11:35:32,852 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Initializing >2014-09-29 11:35:32,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Initializing. >2014-09-29 11:35:32,990 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment setup >2014-09-29 11:35:33,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment setup. >2014-09-29 11:35:33,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment packages setup >2014-09-29 11:35:33,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment packages setup. >2014-09-29 11:35:36,586 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Programs detection >2014-09-29 11:35:36,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Programs detection. >2014-09-29 11:35:36,693 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment customization >2014-09-29 11:35:36,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment customization. >2014-09-29 11:35:36,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Kdump supported >2014-09-29 11:35:36,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Kdump supported. >2014-09-29 11:35:36,955 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140929084350-qb3k03.log' >2014-09-29 11:35:37,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20140929084350-qb3k03.log'. >2014-09-29 11:35:37,123 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host rose07.qa.lab.tlv.redhat.com reports unique id 4C4C4544-0058-5610-8056-B5C04F4D5731 >2014-09-29 11:35:37,168 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0058-5610-8056-B5C04F4D5731 to Host rose07.qa.lab.tlv.redhat.com >2014-09-29 11:35:37,414 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Setup validation >2014-09-29 11:35:37,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Setup validation. >2014-09-29 11:35:37,478 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Hardware supports virtualization >2014-09-29 11:35:37,547 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Hardware supports virtualization. >2014-09-29 11:35:37,570 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Transaction setup >2014-09-29 11:35:37,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Transaction setup. >2014-09-29 11:35:37,613 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-09-29 11:35:37,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-09-29 11:35:37,694 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Package installation >2014-09-29 11:35:37,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Package installation. >2014-09-29 11:35:45,466 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Status: Downloading Packages >2014-09-29 11:35:45,567 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Status: Downloading Packages. >2014-09-29 11:35:45,567 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: ioprocess-0.12.0-2.el6ev.x86_64 >2014-09-29 11:35:45,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: ioprocess-0.12.0-2.el6ev.x86_64. >2014-09-29 11:35:45,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: python-ioprocess-0.12.0-2.el6ev.noarch >2014-09-29 11:35:45,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: python-ioprocess-0.12.0-2.el6ev.noarch. >2014-09-29 11:35:45,728 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-4.16.5-2.el6ev.x86_64 >2014-09-29 11:35:45,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-4.16.5-2.el6ev.x86_64. >2014-09-29 11:35:45,816 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-cli-4.16.5-2.el6ev.noarch >2014-09-29 11:35:45,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-cli-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:45,878 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-jsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:45,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-jsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:45,920 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-python-4.16.5-2.el6ev.noarch >2014-09-29 11:35:45,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-python-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:45,960 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch >2014-09-29 11:35:46,001 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:46,002 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-xmlrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:46,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-xmlrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:46,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Download/Verify: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:46,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Download/Verify: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:46,084 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Status: Check Package Signatures >2014-09-29 11:35:46,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Status: Check Package Signatures. >2014-09-29 11:35:46,123 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Status: Running Test Transaction >2014-09-29 11:35:46,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Status: Running Test Transaction. >2014-09-29 11:35:46,160 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Status: Running Transaction >2014-09-29 11:35:46,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Status: Running Transaction. >2014-09-29 11:35:46,272 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 1/18: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch >2014-09-29 11:35:46,317 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 1/18: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:46,805 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 2/18: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:46,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 2/18: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:47,156 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 3/18: ioprocess-0.12.0-2.el6ev.x86_64 >2014-09-29 11:35:47,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 3/18: ioprocess-0.12.0-2.el6ev.x86_64. >2014-09-29 11:35:47,505 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 4/18: python-ioprocess-0.12.0-2.el6ev.noarch >2014-09-29 11:35:47,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 4/18: python-ioprocess-0.12.0-2.el6ev.noarch. >2014-09-29 11:35:47,807 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 5/18: vdsm-python-4.16.5-2.el6ev.noarch >2014-09-29 11:35:47,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 5/18: vdsm-python-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:48,357 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 6/18: vdsm-xmlrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:48,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 6/18: vdsm-xmlrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:48,663 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 7/18: vdsm-cli-4.16.5-2.el6ev.noarch >2014-09-29 11:35:48,701 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 7/18: vdsm-cli-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:49,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 8/18: vdsm-jsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:35:49,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 8/18: vdsm-jsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:35:49,676 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum update: 9/18: vdsm-4.16.5-2.el6ev.x86_64 >2014-09-29 11:35:49,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum update: 9/18: vdsm-4.16.5-2.el6ev.x86_64. >2014-09-29 11:35:58,036 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 10/18: vdsm >2014-09-29 11:35:58,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 10/18: vdsm. >2014-09-29 11:35:58,739 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 11/18: vdsm-cli >2014-09-29 11:35:58,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 11/18: vdsm-cli. >2014-09-29 11:35:58,973 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 12/18: vdsm-jsonrpc >2014-09-29 11:35:59,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 12/18: vdsm-jsonrpc. >2014-09-29 11:35:59,188 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 13/18: vdsm-xmlrpc >2014-09-29 11:35:59,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 13/18: vdsm-xmlrpc. >2014-09-29 11:35:59,366 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 14/18: vdsm-python >2014-09-29 11:35:59,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 14/18: vdsm-python. >2014-09-29 11:35:59,613 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 15/18: python-ioprocess >2014-09-29 11:35:59,653 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 15/18: python-ioprocess. >2014-09-29 11:35:59,801 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 16/18: vdsm-python-zombiereaper >2014-09-29 11:35:59,839 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 16/18: vdsm-python-zombiereaper. >2014-09-29 11:35:59,976 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 17/18: vdsm-yajsonrpc >2014-09-29 11:36:00,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 17/18: vdsm-yajsonrpc. >2014-09-29 11:36:00,163 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum updated: 18/18: ioprocess >2014-09-29 11:36:00,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum updated: 18/18: ioprocess. >2014-09-29 11:36:01,319 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 1/18: python-ioprocess.noarch 0:0.12.0-2.el6ev - u >2014-09-29 11:36:01,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 1/18: python-ioprocess.noarch 0:0.12.0-2.el6ev - u. >2014-09-29 11:36:01,443 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 2/18: ioprocess.x86_64 0:0.12.0-2.el6ev - u >2014-09-29 11:36:01,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 2/18: ioprocess.x86_64 0:0.12.0-2.el6ev - u. >2014-09-29 11:36:01,526 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 3/18: vdsm-yajsonrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 3/18: vdsm-yajsonrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,604 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 4/18: vdsm.x86_64 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 4/18: vdsm.x86_64 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,695 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 5/18: vdsm-python.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 5/18: vdsm-python.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,771 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 6/18: vdsm-cli.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,820 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 6/18: vdsm-cli.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 7/18: vdsm-xmlrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 7/18: vdsm-xmlrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,919 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 8/18: vdsm-jsonrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:01,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 8/18: vdsm-jsonrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:01,978 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 9/18: vdsm-python-zombiereaper.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:36:02,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 9/18: vdsm-python-zombiereaper.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:36:02,049 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 10/18: python-ioprocess.noarch 0:0.5.0-1.el6 - ud >2014-09-29 11:36:02,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 10/18: python-ioprocess.noarch 0:0.5.0-1.el6 - ud. >2014-09-29 11:36:02,087 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 11/18: vdsm-xmlrpc.noarch 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 11/18: vdsm-xmlrpc.noarch 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,124 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 12/18: vdsm-yajsonrpc.noarch 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 12/18: vdsm-yajsonrpc.noarch 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,161 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 13/18: vdsm-python.x86_64 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 13/18: vdsm-python.x86_64 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 14/18: vdsm-python-zombiereaper.noarch 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 14/18: vdsm-python-zombiereaper.noarch 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,238 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 15/18: vdsm.x86_64 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 15/18: vdsm.x86_64 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,279 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 16/18: ioprocess.x86_64 0:0.5.0-1.el6 - ud >2014-09-29 11:36:02,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 16/18: ioprocess.x86_64 0:0.5.0-1.el6 - ud. >2014-09-29 11:36:02,346 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 17/18: vdsm-jsonrpc.noarch 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,409 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 17/18: vdsm-jsonrpc.noarch 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,409 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Yum Verify: 18/18: vdsm-cli.noarch 0:4.16.3-2.el6 - ud >2014-09-29 11:36:02,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Yum Verify: 18/18: vdsm-cli.noarch 0:4.16.3-2.el6 - ud. >2014-09-29 11:36:02,463 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-09-29 11:36:02,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-09-29 11:36:02,536 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Setting time >2014-09-29 11:36:02,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Setting time. >2014-09-29 11:36:05,074 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Enrolling certificate >2014-09-29 11:36:05,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Enrolling certificate. >2014-09-29 11:36:06,459 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Transaction commit >2014-09-29 11:36:06,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Transaction commit. >2014-09-29 11:36:06,531 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Closing up >2014-09-29 11:36:06,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Closing up. >2014-09-29 11:36:06,912 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Restarting kdump >2014-09-29 11:36:06,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Restarting kdump. >2014-09-29 11:36:07,655 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stopping libvirtd >2014-09-29 11:36:07,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stopping libvirtd. >2014-09-29 11:36:07,868 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Starting vdsm >2014-09-29 11:36:07,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Starting vdsm. >2014-09-29 11:36:09,664 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Pre-termination >2014-09-29 11:36:09,701 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Pre-termination. >2014-09-29 11:36:09,724 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929113609-rose07.qa.lab.tlv.redhat.com-20b19173.log' >2014-09-29 11:36:09,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929113609-rose07.qa.lab.tlv.redhat.com-20b19173.log'. >2014-09-29 11:36:10,052 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Termination >2014-09-29 11:36:10,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 20b19173, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Termination. >2014-09-29 11:36:10,212 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 11:36:10,719 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-41) [20b19173] Engine managed to communicate with VDSM agent on host rose07.qa.lab.tlv.redhat.com >2014-09-29 11:36:11,041 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-41) [20b19173] The management network rhevm is already configured on host rose07.qa.lab.tlv.redhat.com >2014-09-29 11:36:11,074 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10f63bf7 >2014-09-29 11:36:11,079 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] FINISH, SetVdsStatusVDSCommand, log id: 10f63bf7 >2014-09-29 11:36:11,079 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] After Installation host rose07.qa.lab.tlv.redhat.com, VDS >2014-09-29 11:36:11,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-41) [20b19173] Correlation ID: 20b19173, Job ID: ead92c25-0512-4d82-a771-05f1ba85c8a3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com installed >2014-09-29 11:36:11,123 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-41) [20b19173] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS >, sharedLocks= ] >2014-09-29 11:36:12,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 40e7ca58 >2014-09-29 11:36:12,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, GetHardwareInfoVDSCommand, log id: 40e7ca58 >2014-09-29 11:36:12,448 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [7b014c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 11:36:12,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b014c7] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 3ea5785c >2014-09-29 11:36:12,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b014c7] FINISH, GetHardwareInfoVDSCommand, log id: 3ea5785c >2014-09-29 11:36:12,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-58) [7b014c7] START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3], vmIds=[0855adca-3940-4a9d-ac40-63bc25f26c34]), log id: 9e06bc8 >2014-09-29 11:36:12,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-58) [7b014c7] FINISH, FullListVdsCommand, return: [{username=, emulatedMachine=rhel6.5.0, afterMigrationStatus=, spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, pid=28030, devices=[Ljava.lang.Object;@378dc2e4, cpuType=Conroe, smp=2, vmType=kvm, memSize=4096, vmName=HostedEngine, guestFQDN=, exitMessage=User shut down from within the guest, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, displayIp=0, displayPort=-1, guestIPs=, exitCode=0, nicModel=rtl8139,pv, kvmEnable=true, exitReason=7, status=Down, clientIp=, display=vnc, displaySecurePort=-1}], log id: 9e06bc8 >2014-09-29 11:36:12,803 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [7b014c7] Importing VM HostedEngine as HostedEngine, as it is running on the on Host, but does not exist in the engine. >2014-09-29 11:36:12,856 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [7c35079] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:36:13,014 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:36:13,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 751209cb >2014-09-29 11:36:13,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 751209cb >2014-09-29 11:36:13,278 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-29 11:36:13,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 5d67502b >2014-09-29 11:36:13,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, ConnectStoragePoolVDSCommand, log id: 5d67502b >2014-09-29 11:36:13,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 11:36:13,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 11:36:13,826 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 11:36:13,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 43801c76 >2014-09-29 11:36:14,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 43801c76 >2014-09-29 11:36:14,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 6770a1e6 >2014-09-29 11:36:14,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [35dd98fa] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6770a1e6 >2014-09-29 11:36:14,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-09-29 11:36:14,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [35dd98fa] Correlation ID: 7c35079, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-09-29 11:36:14,668 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-58) [36d3f1c6] Lock Acquired to object EngineLock [exclusiveLocks= key: HostedEngine value: VM_NAME >, sharedLocks= ] >2014-09-29 11:36:14,784 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-58) [36d3f1c6] Running command: AddVmFromScratchCommand internal: true. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group CREATE_VM with role type USER >2014-09-29 11:36:15,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [36d3f1c6] Correlation ID: 36d3f1c6, Call Stack: null, Custom Event ID: -1, Message: VM HostedEngine was created by <UNKNOWN>. >2014-09-29 11:36:15,565 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-58) [36d3f1c6] Lock freed to object EngineLock [exclusiveLocks= key: HostedEngine value: VM_NAME >, sharedLocks= ] >2014-09-29 11:36:15,681 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [4a50a139] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 11:36:15,761 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 11:36:18,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-74) START, DestroyVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, force=false, secondsToWait=0, gracefully=false, reason=), log id: 583b8c49 >2014-09-29 11:36:18,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, DestroyVDSCommand, log id: 583b8c49 >2014-09-29 11:43:11,370 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [33c942d5] Running command: AddVdsCommand internal: false. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-29 11:43:11,394 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:43:11,408 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] Lock freed to object EngineLock [exclusiveLocks= key: 43e1738a-3a93-481a-befe-65c12fbc155f value: REGISTER_VDS >, sharedLocks= ] >2014-09-29 11:43:11,411 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] START, RemoveVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 77848522 >2014-09-29 11:43:11,411 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] FINISH, RemoveVdsVDSCommand, log id: 77848522 >2014-09-29 11:43:11,414 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] START, AddVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 44f2977b >2014-09-29 11:43:11,414 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] AddVds - entered , starting logic to add VDS f10f0801-7d59-421d-9d4a-a5fa3b874b13 >2014-09-29 11:43:11,415 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] AddVds - VDS f10f0801-7d59-421d-9d4a-a5fa3b874b13 was added, will try to add it to the resource manager >2014-09-29 11:43:11,416 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [434a15ac] Entered VdsManager constructor >2014-09-29 11:43:11,428 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [434a15ac] Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-09-29 11:43:11,429 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [434a15ac] VDS f10f0801-7d59-421d-9d4a-a5fa3b874b13 was added to the Resource Manager >2014-09-29 11:43:11,430 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [434a15ac] FINISH, AddVdsVDSCommand, log id: 44f2977b >2014-09-29 11:43:11,436 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [434a15ac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 11:43:11,490 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [434a15ac] Correlation ID: 33c942d5, Job ID: e1c5bbd6-fb0c-4bc2-bad3-def1d2e31b01, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was added by admin. >2014-09-29 11:43:11,497 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:43:11,513 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:43:11,514 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Before Installation host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 11:43:11,519 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1c616b31 >2014-09-29 11:43:11,523 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] FINISH, SetVdsStatusVDSCommand, log id: 1c616b31 >2014-09-29 11:43:11,569 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Installation monique-vds01.tlv.redhat.com: Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03 >2014-09-29 11:43:11,596 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03. >2014-09-29 11:43:11,652 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Installation of monique-vds01.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-29 11:43:11,698 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-28) SSH execute root@monique-vds01.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-29 11:43:13,522 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Initializing >2014-09-29 11:43:13,549 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Initializing. >2014-09-29 11:43:13,573 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment setup >2014-09-29 11:43:13,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment setup. >2014-09-29 11:43:13,705 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment packages setup >2014-09-29 11:43:13,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment packages setup. >2014-09-29 11:43:14,596 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Initializing Host: monique-vds01.tlv.redhat.com >2014-09-29 11:43:44,130 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Downloading: epel/primary_db 2.9 M(46%) >2014-09-29 11:43:44,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Downloading: epel/primary_db 2.9 M(46%). >2014-09-29 11:44:14,345 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Downloading: rhel-65-zstream/primary_db 3.5 M(86%) >2014-09-29 11:44:14,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Downloading: rhel-65-zstream/primary_db 3.5 M(86%). >2014-09-29 11:44:19,426 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Programs detection >2014-09-29 11:44:19,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Programs detection. >2014-09-29 11:44:19,462 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment customization >2014-09-29 11:44:19,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment customization. >2014-09-29 11:44:20,355 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Kdump supported >2014-09-29 11:44:20,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Kdump supported. >2014-09-29 11:44:20,415 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140929115212-ou3nar.log' >2014-09-29 11:44:20,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20140929115212-ou3nar.log'. >2014-09-29 11:44:20,533 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds01.tlv.redhat.com reports unique id 44454C4C-5A00-1044-804A-CAC04F32354A >2014-09-29 11:44:20,535 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 44454C4C-5A00-1044-804A-CAC04F32354A to Host monique-vds01.tlv.redhat.com >2014-09-29 11:44:20,633 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Disabling Kdump integration >2014-09-29 11:44:20,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Disabling Kdump integration. >2014-09-29 11:44:20,679 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Setup validation >2014-09-29 11:44:20,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Setup validation. >2014-09-29 11:44:20,743 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Hardware supports virtualization >2014-09-29 11:44:20,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Hardware supports virtualization. >2014-09-29 11:44:21,040 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Transaction setup >2014-09-29 11:44:21,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Transaction setup. >2014-09-29 11:44:21,079 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Misc configuration >2014-09-29 11:44:21,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Misc configuration. >2014-09-29 11:44:21,086 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Package installation >2014-09-29 11:44:21,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Package installation. >2014-09-29 11:44:32,593 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Failed to execute stage 'Package installation': Command '/sbin/service' failed to execute >2014-09-29 11:44:32,617 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host monique-vds01.tlv.redhat.com. Failed to execute stage 'Package installation': Command '/sbin/service' failed to execute. >2014-09-29 11:44:32,617 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Performing yum transaction rollback >2014-09-29 11:44:32,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Performing yum transaction rollback. >2014-09-29 11:44:32,641 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Pre-termination >2014-09-29 11:44:32,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Pre-termination. >2014-09-29 11:44:32,729 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929114432-monique-vds01.tlv.redhat.com-257f63a4.log' >2014-09-29 11:44:32,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929114432-monique-vds01.tlv.redhat.com-257f63a4.log'. >2014-09-29 11:44:32,940 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Termination >2014-09-29 11:44:32,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 257f63a4, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Termination. >2014-09-29 11:44:33,098 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-28) SSH error running command root@monique-vds01.tlv.redhat.com:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2261) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:203) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:44:33,101 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Error during host monique-vds01.tlv.redhat.com install: java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2261) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:203) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:44:33,104 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Host installation failed for host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com.: java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2261) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:203) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:44:33,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 39ff6960 >2014-09-29 11:44:33,138 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] FINISH, SetVdsStatusVDSCommand, log id: 39ff6960 >2014-09-29 11:44:33,151 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Correlation ID: 257f63a4, Job ID: e1c5bbd6-fb0c-4bc2-bad3-def1d2e31b01, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com'. >2014-09-29 11:44:33,158 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-28) [257f63a4] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:47:31,164 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41:SD1 in problem. vds: monique-vds05 >2014-09-29 11:47:34,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-81) Command ListVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-09-29 11:47:34,213 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host monique-vds05 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 11:47:34,226 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 11:47:34,226 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy124.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:47:34,237 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Failed to refresh VDS , vds = eec426a9-0846-4c02-83f0-c6fdb9566de0 : monique-vds05, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-09-29 11:47:36,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) Command SpmStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 11:47:36,358 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:47:36,370 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-29 11:47:36,381 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Correlation ID: 1aafc5c7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds05, Error: Network error during communication with the Host.). >2014-09-29 11:47:36,392 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-09-29 11:47:36,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Command SpmStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 11:47:36,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Irs placed on server eec426a9-0846-4c02-83f0-c6fdb9566de0 failed. Proceed Failover >2014-09-29 11:47:36,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-09-29 11:47:36,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-09-29 11:47:36,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65db430b >2014-09-29 11:47:36,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] spmStart polling started: taskId = 9122ccbf-7f77-4b4d-8cb7-332cbbf6afbf >2014-09-29 11:47:37,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 57fe2ce7 >2014-09-29 11:47:37,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] spmStart polling ended: taskId = 9122ccbf-7f77-4b4d-8cb7-332cbbf6afbf task status = finished >2014-09-29 11:47:37,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GetHardwareInfoVDSCommand, log id: 57fe2ce7 >2014-09-29 11:47:37,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] spmStart polling ended, spm status: SPM >2014-09-29 11:47:37,585 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [7fcb00d9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 11:47:37,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, taskId=9122ccbf-7f77-4b4d-8cb7-332cbbf6afbf), log id: 7d892073 >2014-09-29 11:47:37,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] FINISH, HSMClearTaskVDSCommand, log id: 7d892073 >2014-09-29 11:47:37,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b6d8ecd, log id: 65db430b >2014-09-29 11:47:37,687 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [5ce1a178] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:47:37,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-09-29 11:47:37,698 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:47:37,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] START, ConnectStorageServerVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 14bfe2a5 >2014-09-29 11:47:37,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [1aafc5c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-09-29 11:47:37,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 14bfe2a5 >2014-09-29 11:47:37,820 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] Host monique-vds05 storage connection was succeeded >2014-09-29 11:47:37,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: 2568e543 >2014-09-29 11:47:37,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-41) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vdsId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 4dbe251f >2014-09-29 11:47:37,935 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 11:47:37,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 4dbe251f >2014-09-29 11:47:37,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 31f3967a >2014-09-29 11:47:38,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 31f3967a >2014-09-29 11:47:38,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05 was set to Up. >2014-09-29 11:47:38,049 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-29 11:47:38,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [1b7afe7c] Correlation ID: 5ce1a178, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 11:47:38,061 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [7f27cea2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 11:47:38,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-45) Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41:SD1 recovered from problem. vds: monique-vds05 >2014-09-29 11:47:38,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-45) Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-09-29 11:47:38,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-29 11:47:38,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 6a055a01 >2014-09-29 11:47:38,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a055a01 >2014-09-29 11:47:38,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2568e543 >2014-09-29 11:47:38,142 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-30) [1aafc5c7] Discovered no tasks on Storage Pool DC1 >2014-09-29 11:54:14,392 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:54:14,392 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:54:14,393 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-6) [283eee5e] Running command: InstallVdsCommand internal: false. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-29 11:54:14,633 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:54:14,633 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:54:14,636 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (ajp-/127.0.0.1:8702-6) [283eee5e] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:54:14,639 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:54:14,639 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Before Installation host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 11:54:14,643 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1c6f4996 >2014-09-29 11:54:14,660 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [283eee5e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 11:54:14,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] FINISH, SetVdsStatusVDSCommand, log id: 1c6f4996 >2014-09-29 11:54:14,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [283eee5e] Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com configuration was updated by admin. >2014-09-29 11:54:14,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Installation monique-vds01.tlv.redhat.com: Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03 >2014-09-29 11:54:14,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03. >2014-09-29 11:54:15,268 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Installation of monique-vds01.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-29 11:54:15,278 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-25) SSH execute root@monique-vds01.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-29 11:54:16,738 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Initializing >2014-09-29 11:54:16,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Initializing. >2014-09-29 11:54:16,768 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment setup >2014-09-29 11:54:16,779 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment setup. >2014-09-29 11:54:16,786 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment packages setup >2014-09-29 11:54:16,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment packages setup. >2014-09-29 11:54:17,448 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Cannot queue package iproute: Cannot retrieve repository metadata (repomd.xml) for repository: JBoss_latest. Please verify its path and try again >2014-09-29 11:54:17,454 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host monique-vds01.tlv.redhat.com. Yum Cannot queue package iproute: Cannot retrieve repository metadata (repomd.xml) for repository: JBoss_latest. Please verify its path and try again. >2014-09-29 11:54:17,475 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Failed to execute stage 'Environment packages setup': Cannot retrieve repository metadata (repomd.xml) for repository: JBoss_latest. Please verify its path and try again >2014-09-29 11:54:17,481 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host monique-vds01.tlv.redhat.com. Failed to execute stage 'Environment packages setup': Cannot retrieve repository metadata (repomd.xml) for repository: JBoss_latest. Please verify its path and try again. >2014-09-29 11:54:17,481 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Performing yum transaction rollback >2014-09-29 11:54:17,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Performing yum transaction rollback. >2014-09-29 11:54:17,503 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Pre-termination >2014-09-29 11:54:17,518 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Pre-termination. >2014-09-29 11:54:17,609 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929115417-monique-vds01.tlv.redhat.com-283eee5e.log' >2014-09-29 11:54:17,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929115417-monique-vds01.tlv.redhat.com-283eee5e.log'. >2014-09-29 11:54:17,758 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Termination >2014-09-29 11:54:17,769 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Termination. >2014-09-29 11:54:17,855 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-25) SSH error running command root@monique-vds01.tlv.redhat.com:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:191) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:158) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:167) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:54:17,856 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Error during host monique-vds01.tlv.redhat.com install: java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:191) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:158) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:167) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:54:17,860 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Host installation failed for host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com.: java.io.IOException: Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] > at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:318) [uutils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1118) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:151) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:191) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:158) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:167) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-09-29 11:54:17,887 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 375fb697 >2014-09-29 11:54:17,912 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] FINISH, SetVdsStatusVDSCommand, log id: 375fb697 >2014-09-29 11:54:17,938 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Correlation ID: 283eee5e, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root@monique-vds01.tlv.redhat.com'. >2014-09-29 11:54:17,939 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-25) [283eee5e] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:55:44,737 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:55:44,737 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:55:44,738 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-7) [78f045f1] Running command: InstallVdsCommand internal: false. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-29 11:55:44,776 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:55:44,776 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 11:55:44,828 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (ajp-/127.0.0.1:8702-7) [78f045f1] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:55:44,830 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:55:44,831 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Before Installation host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 11:55:44,834 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4df9aa25 >2014-09-29 11:55:44,853 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [78f045f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 11:55:44,857 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] FINISH, SetVdsStatusVDSCommand, log id: 4df9aa25 >2014-09-29 11:55:44,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [78f045f1] Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com configuration was updated by admin. >2014-09-29 11:55:44,955 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Installation monique-vds01.tlv.redhat.com: Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03 >2014-09-29 11:55:44,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03. >2014-09-29 11:55:45,038 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Installation of monique-vds01.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-29 11:55:45,047 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-33) SSH execute root@monique-vds01.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-29 11:55:46,529 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Initializing >2014-09-29 11:55:46,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Initializing. >2014-09-29 11:55:46,560 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment setup >2014-09-29 11:55:46,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment setup. >2014-09-29 11:55:46,588 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment packages setup >2014-09-29 11:55:46,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment packages setup. >2014-09-29 11:56:06,640 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Programs detection >2014-09-29 11:56:06,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Programs detection. >2014-09-29 11:56:06,647 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment customization >2014-09-29 11:56:06,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment customization. >2014-09-29 11:56:07,044 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Kdump supported >2014-09-29 11:56:07,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Kdump supported. >2014-09-29 11:56:07,062 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140929120446-n6rjmx.log' >2014-09-29 11:56:07,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20140929120446-n6rjmx.log'. >2014-09-29 11:56:07,150 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds01.tlv.redhat.com reports unique id 44454C4C-5A00-1044-804A-CAC04F32354A >2014-09-29 11:56:07,153 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 44454C4C-5A00-1044-804A-CAC04F32354A to Host monique-vds01.tlv.redhat.com >2014-09-29 11:56:07,321 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Disabling Kdump integration >2014-09-29 11:56:07,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Disabling Kdump integration. >2014-09-29 11:56:07,365 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Setup validation >2014-09-29 11:56:07,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Setup validation. >2014-09-29 11:56:07,422 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Hardware supports virtualization >2014-09-29 11:56:07,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Hardware supports virtualization. >2014-09-29 11:56:07,713 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Transaction setup >2014-09-29 11:56:07,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Transaction setup. >2014-09-29 11:56:07,744 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Misc configuration >2014-09-29 11:56:07,771 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Misc configuration. >2014-09-29 11:56:07,772 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Package installation >2014-09-29 11:56:07,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Package installation. >2014-09-29 11:56:27,015 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Status: Downloading Packages >2014-09-29 11:56:27,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Status: Downloading Packages. >2014-09-29 11:56:27,028 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: ioprocess-0.12.0-2.el6ev.x86_64 >2014-09-29 11:56:27,036 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: ioprocess-0.12.0-2.el6ev.x86_64. >2014-09-29 11:56:27,036 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: python-ioprocess-0.12.0-2.el6ev.noarch >2014-09-29 11:56:27,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: python-ioprocess-0.12.0-2.el6ev.noarch. >2014-09-29 11:56:27,044 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-4.16.5-2.el6ev.x86_64 >2014-09-29 11:56:27,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-4.16.5-2.el6ev.x86_64. >2014-09-29 11:56:27,053 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-cli-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-cli-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,060 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-jsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,066 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-jsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,066 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-python-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-python-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,074 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,079 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,079 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-xmlrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,085 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-xmlrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,085 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Download/Verify: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:27,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Download/Verify: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:27,090 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Status: Check Package Signatures >2014-09-29 11:56:27,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Status: Check Package Signatures. >2014-09-29 11:56:27,095 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Status: Running Test Transaction >2014-09-29 11:56:27,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Status: Running Test Transaction. >2014-09-29 11:56:27,962 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Status: Running Transaction >2014-09-29 11:56:27,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Status: Running Transaction. >2014-09-29 11:56:28,624 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 1/18: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch >2014-09-29 11:56:28,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 1/18: vdsm-python-zombiereaper-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:29,326 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 2/18: ioprocess-0.12.0-2.el6ev.x86_64 >2014-09-29 11:56:29,332 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 2/18: ioprocess-0.12.0-2.el6ev.x86_64. >2014-09-29 11:56:29,905 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 3/18: python-ioprocess-0.12.0-2.el6ev.noarch >2014-09-29 11:56:29,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 3/18: python-ioprocess-0.12.0-2.el6ev.noarch. >2014-09-29 11:56:30,369 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 4/18: vdsm-python-4.16.5-2.el6ev.noarch >2014-09-29 11:56:30,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 4/18: vdsm-python-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:31,563 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 5/18: vdsm-xmlrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:31,570 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 5/18: vdsm-xmlrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:32,719 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 6/18: vdsm-cli-4.16.5-2.el6ev.noarch >2014-09-29 11:56:32,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 6/18: vdsm-cli-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:33,330 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 7/18: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:33,336 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 7/18: vdsm-yajsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:33,753 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 8/18: vdsm-jsonrpc-4.16.5-2.el6ev.noarch >2014-09-29 11:56:33,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 8/18: vdsm-jsonrpc-4.16.5-2.el6ev.noarch. >2014-09-29 11:56:34,645 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum update: 9/18: vdsm-4.16.5-2.el6ev.x86_64 >2014-09-29 11:56:34,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum update: 9/18: vdsm-4.16.5-2.el6ev.x86_64. >2014-09-29 11:56:52,764 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 10/18: vdsm >2014-09-29 11:56:52,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 10/18: vdsm. >2014-09-29 11:56:54,173 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 11/18: vdsm-cli >2014-09-29 11:56:54,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 11/18: vdsm-cli. >2014-09-29 11:56:54,466 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 12/18: vdsm-jsonrpc >2014-09-29 11:56:54,473 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 12/18: vdsm-jsonrpc. >2014-09-29 11:56:54,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 13/18: vdsm-xmlrpc >2014-09-29 11:56:54,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 13/18: vdsm-xmlrpc. >2014-09-29 11:56:55,031 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 14/18: vdsm-python >2014-09-29 11:56:55,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 14/18: vdsm-python. >2014-09-29 11:56:55,498 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 15/18: python-ioprocess >2014-09-29 11:56:55,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 15/18: python-ioprocess. >2014-09-29 11:56:55,818 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 16/18: vdsm-python-zombiereaper >2014-09-29 11:56:55,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 16/18: vdsm-python-zombiereaper. >2014-09-29 11:56:56,044 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 17/18: vdsm-yajsonrpc >2014-09-29 11:56:56,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 17/18: vdsm-yajsonrpc. >2014-09-29 11:56:56,288 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum updated: 18/18: ioprocess >2014-09-29 11:56:56,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum updated: 18/18: ioprocess. >2014-09-29 11:56:58,781 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 1/18: python-ioprocess.noarch 0:0.12.0-2.el6ev - u >2014-09-29 11:56:58,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 1/18: python-ioprocess.noarch 0:0.12.0-2.el6ev - u. >2014-09-29 11:56:58,905 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 2/18: vdsm-yajsonrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:58,911 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 2/18: vdsm-yajsonrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,033 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 3/18: vdsm.x86_64 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 3/18: vdsm.x86_64 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,129 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 4/18: vdsm-python.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 4/18: vdsm-python.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,287 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 5/18: vdsm-cli.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 5/18: vdsm-cli.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,404 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 6/18: vdsm-xmlrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 6/18: vdsm-xmlrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,486 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 7/18: vdsm-python-zombiereaper.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 7/18: vdsm-python-zombiereaper.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,606 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 8/18: vdsm-jsonrpc.noarch 0:4.16.5-2.el6ev - u >2014-09-29 11:56:59,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 8/18: vdsm-jsonrpc.noarch 0:4.16.5-2.el6ev - u. >2014-09-29 11:56:59,702 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 9/18: ioprocess.x86_64 0:0.12.0-2.el6ev - u >2014-09-29 11:56:59,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 9/18: ioprocess.x86_64 0:0.12.0-2.el6ev - u. >2014-09-29 11:56:59,907 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 10/18: vdsm-python.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 10/18: vdsm-python.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,914 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 11/18: python-ioprocess.noarch 0:0.5.0-1.el6 - ud >2014-09-29 11:56:59,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 11/18: python-ioprocess.noarch 0:0.5.0-1.el6 - ud. >2014-09-29 11:56:59,921 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 12/18: ioprocess.x86_64 0:0.5.0-1.el6 - ud >2014-09-29 11:56:59,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 12/18: ioprocess.x86_64 0:0.5.0-1.el6 - ud. >2014-09-29 11:56:59,929 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 13/18: vdsm-cli.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 13/18: vdsm-cli.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,938 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 14/18: vdsm-jsonrpc.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 14/18: vdsm-jsonrpc.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,943 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 15/18: vdsm.x86_64 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 15/18: vdsm.x86_64 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,949 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 16/18: vdsm-xmlrpc.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 16/18: vdsm-xmlrpc.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,955 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 17/18: vdsm-python-zombiereaper.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 17/18: vdsm-python-zombiereaper.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:56:59,960 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Yum Verify: 18/18: vdsm-yajsonrpc.noarch 0:4.16.3-3.el6ev.beta - ud >2014-09-29 11:56:59,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Yum Verify: 18/18: vdsm-yajsonrpc.noarch 0:4.16.3-3.el6ev.beta - ud. >2014-09-29 11:57:00,214 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Misc configuration >2014-09-29 11:57:00,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Misc configuration. >2014-09-29 11:57:02,500 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Enrolling certificate >2014-09-29 11:57:02,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Enrolling certificate. >2014-09-29 11:57:03,935 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Transaction commit >2014-09-29 11:57:03,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Transaction commit. >2014-09-29 11:57:04,072 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Closing up >2014-09-29 11:57:04,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Closing up. >2014-09-29 11:57:06,029 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stopping libvirtd >2014-09-29 11:57:06,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stopping libvirtd. >2014-09-29 11:57:06,717 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Starting vdsm >2014-09-29 11:57:06,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Starting vdsm. >2014-09-29 11:57:10,372 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Pre-termination >2014-09-29 11:57:10,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Pre-termination. >2014-09-29 11:57:10,468 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929115710-monique-vds01.tlv.redhat.com-78f045f1.log' >2014-09-29 11:57:10,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929115710-monique-vds01.tlv.redhat.com-78f045f1.log'. >2014-09-29 11:57:10,807 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Termination >2014-09-29 11:57:10,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Termination. >2014-09-29 11:57:11,002 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-09-29 11:57:11,003 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 11:57:11,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 11:57:11,004 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (org.ovirt.thread.pool-7-thread-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:917) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:913) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:405) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:394) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 11:57:11,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-09-29 11:57:11,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Command PollVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-09-29 11:57:11,522 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-09-29 11:57:11,762 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Engine managed to communicate with VDSM agent on host monique-vds01.tlv.redhat.com >2014-09-29 11:57:12,689 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-33) [78f045f1] The management network rhevm is already configured on host monique-vds01.tlv.redhat.com >2014-09-29 11:57:12,692 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1f10241a >2014-09-29 11:57:12,717 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] FINISH, SetVdsStatusVDSCommand, log id: 1f10241a >2014-09-29 11:57:12,717 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] After Installation host monique-vds01.tlv.redhat.com, VDS >2014-09-29 11:57:12,754 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Correlation ID: 78f045f1, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com installed >2014-09-29 11:57:12,754 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [78f045f1] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS >, sharedLocks= ] >2014-09-29 11:57:15,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 6aad1bbd >2014-09-29 11:57:15,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 6aad1bbd >2014-09-29 11:57:15,776 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [4ed6f090] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:57:15,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ed6f090] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 4441d7c6 >2014-09-29 11:57:15,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [4ed6f090] FINISH, GetHardwareInfoVDSCommand, log id: 4441d7c6 >2014-09-29 11:57:15,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-6) [4ed6f090] START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13], vmIds=[0855adca-3940-4a9d-ac40-63bc25f26c34]), log id: 2d82b2cd >2014-09-29 11:57:16,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-6) [4ed6f090] FINISH, FullListVdsCommand, return: [{status=Up, guestFQDN=, emulatedMachine=rhel6.5.0, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, pid=20115, vmName=HostedEngine, devices=[Ljava.lang.Object;@5fa8c812, nicModel=rtl8139,pv, username=Unknown, vmType=kvm, displayIp=0, display=vnc, displaySecurePort=-1, memSize=4096, displayPort=5900, cpuType=Conroe, guestIPs=, smp=2, pauseCode=NOERR, clientIp=}], log id: 2d82b2cd >2014-09-29 11:57:16,036 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [4ed6f090] Received a console Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {alias=console0, specParams={}, deviceType=console, deviceId=5f4781ed-1bf3-4b50-9867-d4bf79cd974c, device=console, type=console} >2014-09-29 11:57:16,036 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [4ed6f090] Received a memballoon Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=memballoon, specParams={model=none}, deviceType=balloon, type=balloon, alias=balloon0} >2014-09-29 11:57:16,036 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [4ed6f090] Received a vnc Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=vnc, specParams={spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0}, deviceType=graphics, type=graphics, port=5900} >2014-09-29 11:57:16,037 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [4ed6f090] Empty or NULL values were passed for a VM 0855adca-3940-4a9d-ac40-63bc25f26c34 device, Device is skipped >2014-09-29 11:57:16,040 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-6) [2e3cdbe9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:57:16,049 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 11:57:16,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4caa7d37 >2014-09-29 11:57:16,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 4caa7d37 >2014-09-29 11:57:16,793 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-09-29 11:57:16,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-11) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vdsId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 2aecc50c >2014-09-29 11:57:17,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-11) FINISH, ConnectStoragePoolVDSCommand, log id: 2aecc50c >2014-09-29 11:57:17,338 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com has time-drift of 540 seconds while maximum configured value is 300 seconds. >2014-09-29 11:57:17,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 6511a906 >2014-09-29 11:57:17,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-6) [4dbfb058] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6511a906 >2014-09-29 11:57:17,449 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [4dbfb058] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-09-29 11:57:17,453 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [4dbfb058] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 11:57:17,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [4dbfb058] Correlation ID: 2e3cdbe9, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 11:57:17,526 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [41879aa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 11:57:17,532 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [3d142e42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 12:04:30,173 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) VM HostedEngine 0855adca-3940-4a9d-ac40-63bc25f26c34 moved from Up --> PoweringDown >2014-09-29 12:04:42,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-2) START, DestroyVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, force=false, secondsToWait=0, gracefully=false, reason=), log id: 25715638 >2014-09-29 12:04:42,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, DestroyVDSCommand, log id: 25715638 >2014-09-29 12:04:42,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM HostedEngine is down. Exit message: Admin shut down from the engine >2014-09-29 12:04:42,461 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) VM HostedEngine (0855adca-3940-4a9d-ac40-63bc25f26c34) is running in db and not running in VDS monique-vds01.tlv.redhat.com >2014-09-29 12:04:42,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-2) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13], vmIds=[0855adca-3940-4a9d-ac40-63bc25f26c34]), log id: 51da32b >2014-09-29 12:04:42,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-2) FINISH, FullListVdsCommand, return: [], log id: 51da32b >2014-09-29 12:04:42,502 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-7-thread-14) [ea7a8c2] Running command: ProcessDownVmCommand internal: true. >2014-09-29 12:05:12,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13], vmIds=[0855adca-3940-4a9d-ac40-63bc25f26c34]), log id: 48e042fa >2014-09-29 12:05:12,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FullListVdsCommand, return: [{status=WaitForLaunch, nicModel=rtl8139,pv, emulatedMachine=rhel6.5.0, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, pid=0, vmName=HostedEngine, devices=[Ljava.lang.Object;@7b4d5076, smp=2, vmType=kvm, display=vnc, displaySecurePort=-1, memSize=4096, displayPort=-1, cpuType=Conroe, spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0, clientIp=}], log id: 48e042fa >2014-09-29 12:05:12,850 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Received a console Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {alias=console0, specParams={}, deviceType=console, deviceId=5f4781ed-1bf3-4b50-9867-d4bf79cd974c, device=console, type=console} >2014-09-29 12:05:12,850 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Received a memballoon Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=memballoon, specParams={model=none}, deviceType=balloon, type=balloon} >2014-09-29 12:05:12,850 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Received a scsi Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=scsi, deviceType=controller, model=virtio-scsi, type=controller} >2014-09-29 12:05:12,850 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Received a vnc Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=vnc, specParams={spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0}, deviceType=graphics, type=graphics} >2014-09-29 12:05:15,897 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) VM HostedEngine 0855adca-3940-4a9d-ac40-63bc25f26c34 moved from WaitForLaunch --> PoweringUp >2014-09-29 12:05:15,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-19) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13], vmIds=[0855adca-3940-4a9d-ac40-63bc25f26c34]), log id: c0212c2 >2014-09-29 12:05:15,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-19) FINISH, FullListVdsCommand, return: [{status=Up, nicModel=rtl8139,pv, emulatedMachine=rhel6.5.0, vmId=0855adca-3940-4a9d-ac40-63bc25f26c34, pid=6121, vmName=HostedEngine, devices=[Ljava.lang.Object;@5c1d84b6, smp=2, vmType=kvm, display=vnc, displaySecurePort=-1, memSize=4096, displayPort=5900, cpuType=Conroe, spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0, clientIp=, pauseCode=NOERR}], log id: c0212c2 >2014-09-29 12:05:15,916 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Received a console Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {alias=console0, specParams={}, deviceType=console, deviceId=5f4781ed-1bf3-4b50-9867-d4bf79cd974c, device=console, type=console} >2014-09-29 12:05:15,916 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Received a memballoon Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=memballoon, specParams={model=none}, deviceType=balloon, type=balloon, alias=balloon0} >2014-09-29 12:05:15,917 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Received a vnc Device without an address when processing VM 0855adca-3940-4a9d-ac40-63bc25f26c34 devices, skipping device: {device=vnc, specParams={spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0}, deviceType=graphics, type=graphics, port=5900} >2014-09-29 12:05:15,917 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Empty or NULL values were passed for a VM 0855adca-3940-4a9d-ac40-63bc25f26c34 device, Device is skipped >2014-09-29 12:06:14,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) VM HostedEngine 0855adca-3940-4a9d-ac40-63bc25f26c34 moved from PoweringUp --> Up >2014-09-29 12:07:43,415 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-09-29 12:07:43,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-09-29 12:08:11,926 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 540 seconds while maximum configured value is 300 seconds. >2014-09-29 12:08:24,004 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-46) [233b2f2b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 12:08:24,020 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) [233b2f2b] START, SetVdsStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 495a7620 >2014-09-29 12:08:24,046 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) [233b2f2b] FINISH, SetVdsStatusVDSCommand, log id: 495a7620 >2014-09-29 12:08:24,134 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = eec426a9-0846-4c02-83f0-c6fdb9566de0 : monique-vds05 >2014-09-29 12:08:24,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) Clearing cache of pool: 43e1738a-3a93-481a-befe-65c12fbc155f for problematic entities of VDS: monique-vds05. >2014-09-29 12:08:24,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) Removing vds [eec426a9-0846-4c02-83f0-c6fdb9566de0] from the domain in maintenance cache >2014-09-29 12:08:24,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) Removing host(s) [eec426a9-0846-4c02-83f0-c6fdb9566de0] from hosts unseen domain report cache >2014-09-29 12:08:24,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, vds_spm_id = 2), log id: 30144775 >2014-09-29 12:08:24,352 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-46) [233b2f2b] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 12:08:24,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [233b2f2b] Correlation ID: 233b2f2b, Job ID: 521ad12c-84cf-4a1c-b59a-c7e77856f80a, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 was switched to Maintenance mode by admin. >2014-09-29 12:08:28,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, DisconnectStoragePoolVDSCommand, log id: 30144775 >2014-09-29 12:08:28,032 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 12:08:28,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ba9958 >2014-09-29 12:08:28,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] FINISH, DisconnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 7ba9958 >2014-09-29 12:08:29,488 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 12:08:29,488 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 12:08:29,488 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-8) [426522ea] Running command: InstallVdsCommand internal: false. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-29 12:08:29,507 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 12:08:29,508 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 12:08:29,510 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (ajp-/127.0.0.1:8702-8) [426522ea] Lock Acquired to object EngineLock [exclusiveLocks= key: eec426a9-0846-4c02-83f0-c6fdb9566de0 value: VDS >, sharedLocks= ] >2014-09-29 12:08:29,512 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 12:08:29,512 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] Before Installation host eec426a9-0846-4c02-83f0-c6fdb9566de0, monique-vds05 >2014-09-29 12:08:29,529 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [426522ea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-29 12:08:29,539 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] START, SetVdsStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5e370721 >2014-09-29 12:08:29,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [426522ea] Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 configuration was updated by admin. >2014-09-29 12:08:29,544 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] FINISH, SetVdsStatusVDSCommand, log id: 5e370721 >2014-09-29 12:08:29,677 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-13) [426522ea] Installation monique-vds05.tlv.redhat.com: Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: c8:8c:ff:bf:60:67:ae:6a:32:6a:b5:32:62:69:d8:30 >2014-09-29 12:08:29,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [426522ea] Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: c8:8c:ff:bf:60:67:ae:6a:32:6a:b5:32:62:69:d8:30. >2014-09-29 12:08:30,051 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-13) [426522ea] Installation of monique-vds05.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-29 12:08:30,061 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-13) SSH execute root@monique-vds05.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-29 12:08:30,756 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Initializing >2014-09-29 12:08:30,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Initializing. >2014-09-29 12:08:30,806 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment setup >2014-09-29 12:08:30,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment setup. >2014-09-29 12:08:30,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment packages setup >2014-09-29 12:08:30,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment packages setup. >2014-09-29 12:08:34,911 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Programs detection >2014-09-29 12:08:34,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Programs detection. >2014-09-29 12:08:34,928 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment customization >2014-09-29 12:08:34,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Environment customization. >2014-09-29 12:08:35,182 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Kdump unsupported >2014-09-29 12:08:35,189 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Kdump unsupported. >2014-09-29 12:08:35,199 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140929121731-6bv9uq.log' >2014-09-29 12:08:35,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Logs at host located at: '/tmp/ovirt-host-deploy-20140929121731-6bv9uq.log'. >2014-09-29 12:08:35,240 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds05.tlv.redhat.com reports unique id 4C4C4544-0032-4310-8051-B7C04F39354A >2014-09-29 12:08:35,243 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0032-4310-8051-B7C04F39354A to Host monique-vds05.tlv.redhat.com >2014-09-29 12:08:35,325 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Disabling Kdump integration >2014-09-29 12:08:35,332 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Disabling Kdump integration. >2014-09-29 12:08:35,346 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Setup validation >2014-09-29 12:08:35,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Setup validation. >2014-09-29 12:08:35,387 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Hardware supports virtualization >2014-09-29 12:08:35,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Hardware supports virtualization. >2014-09-29 12:08:35,550 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction setup >2014-09-29 12:08:35,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Transaction setup. >2014-09-29 12:08:35,571 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-09-29 12:08:35,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Misc configuration. >2014-09-29 12:08:35,578 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Package installation >2014-09-29 12:08:35,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Package installation. >2014-09-29 12:08:46,809 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-09-29 12:08:46,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Misc configuration. >2014-09-29 12:08:57,391 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Enrolling certificate >2014-09-29 12:08:57,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Enrolling certificate. >2014-09-29 12:08:58,610 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction commit >2014-09-29 12:08:58,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Transaction commit. >2014-09-29 12:08:58,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Closing up >2014-09-29 12:08:58,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Closing up. >2014-09-29 12:08:59,759 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stopping libvirtd >2014-09-29 12:08:59,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stopping libvirtd. >2014-09-29 12:08:59,821 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Starting vdsm >2014-09-29 12:08:59,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Starting vdsm. >2014-09-29 12:09:01,421 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Pre-termination >2014-09-29 12:09:01,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Pre-termination. >2014-09-29 12:09:01,560 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929120901-monique-vds05.tlv.redhat.com-426522ea.log' >2014-09-29 12:09:01,588 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140929120901-monique-vds05.tlv.redhat.com-426522ea.log'. >2014-09-29 12:09:01,988 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Termination >2014-09-29 12:09:02,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05. Stage: Termination. >2014-09-29 12:09:02,209 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-13) [426522ea] Engine managed to communicate with VDSM agent on host monique-vds05 >2014-09-29 12:09:02,519 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-13) [426522ea] The management network rhevm is already configured on host monique-vds05 >2014-09-29 12:09:02,522 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] START, SetVdsStatusVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1494af97 >2014-09-29 12:09:02,551 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] FINISH, SetVdsStatusVDSCommand, log id: 1494af97 >2014-09-29 12:09:02,551 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] After Installation host monique-vds05, VDS >2014-09-29 12:09:02,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-13) [426522ea] Correlation ID: 426522ea, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 installed >2014-09-29 12:09:02,594 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-13) [426522ea] Lock freed to object EngineLock [exclusiveLocks= key: eec426a9-0846-4c02-83f0-c6fdb9566de0 value: VDS >, sharedLocks= ] >2014-09-29 12:09:04,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 468b3026 >2014-09-29 12:09:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, GetHardwareInfoVDSCommand, log id: 468b3026 >2014-09-29 12:09:04,419 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [4e4c8dd7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 12:09:04,448 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [36da812f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 12:09:04,475 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 12:09:04,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] START, ConnectStorageServerVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 66cdb083 >2014-09-29 12:09:05,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 66cdb083 >2014-09-29 12:09:05,018 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] Host monique-vds05 storage connection was succeeded >2014-09-29 12:09:05,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-45) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vdsId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 437f20a6 >2014-09-29 12:09:05,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 437f20a6 >2014-09-29 12:09:05,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 5092e6bd >2014-09-29 12:09:05,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6e3bc963] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5092e6bd >2014-09-29 12:09:05,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6e3bc963] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05 was set to Up. >2014-09-29 12:09:05,345 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6e3bc963] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05. >2014-09-29 12:09:05,357 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6e3bc963] Correlation ID: 36da812f, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 12:09:05,389 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [462f5c31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 12:11:49,677 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetStatsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:49,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-21) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:49,678 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:11:49,696 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:11:49,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:11:49,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:11:49,746 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:49,800 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:11:49,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) vds::refreshVdsStats Failed getVdsStats, vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, error = VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:49,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand.executeVdsBrokerCommand(GetStatsVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:695) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:11:49,828 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:51,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) Command SpmStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:11:51,322 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 12:11:51,329 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-36) [4270fff5] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-29 12:11:51,452 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4270fff5] Correlation ID: 4270fff5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host rose07.qa.lab.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-09-29 12:12:45,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 12:12:45,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-9) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host >2014-09-29 12:12:45,842 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 12:12:45,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-9) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 6f0abc33 >2014-09-29 12:12:46,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6f0abc33 >2014-09-29 12:14:51,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] ERROR, org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand, exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues', log id: 35f6c90a: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand.executeIrsBrokerCommand(GetStoragePoolInfoVDSCommand.java:32) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:156) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:14:52,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@449e6c47 >2014-09-29 12:14:52,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:14:52,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:14:52,870 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:14:52,898 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:14:52,898 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:14:52,912 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:14:53,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@43631b48 >2014-09-29 12:14:53,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:14:53,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:14:53,364 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:14:53,458 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:14:53,458 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:14:53,470 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:01,032 WARN [org.ovirt.engine.core.bll.ShutdownVmCommand] (ajp-/127.0.0.1:8702-1) [5a8e8db8] CanDoAction of action ShutdownVm failed. Reasons:VAR__ACTION__SHUTDOWN,VAR__TYPE__VM,ACTION_TYPE_FAILED_CANNOT_RUN_ACTION_ON_NON_MANAGED_VM >2014-09-29 12:17:17,452 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-7) [1fc58da9] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:17:17,453 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-20) [1fc58da9] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 12:17:17,480 WARN [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-7-thread-20) [6e8050bd] CanDoAction of action ClearNonResponsiveVdsVms failed. Reasons:VDS_CANNOT_CLEAR_VMS_WRONG_STATUS >2014-09-29 12:17:17,516 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-20) [6e8050bd] Correlation ID: 1fc58da9, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host monique-vds01.tlv.redhat.com was started. >2014-09-29 12:17:17,517 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-20) [6e8050bd] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:17:51,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] Command SpmStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:51,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [4270fff5] Irs placed on server 08c6d6dd-82dd-41e7-9e92-6957151a40e3 failed. Proceed Failover >2014-09-29 12:17:52,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [4270fff5] hostFromVds::selectedVds - monique-vds05, spmStatus Free, storage pool DC1 >2014-09-29 12:17:52,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [4270fff5] starting spm on vds monique-vds05, storage pool DC1, prevId -1, LVER -1 >2014-09-29 12:17:52,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] START, SpmStartVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5bc73179 >2014-09-29 12:17:52,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] spmStart polling started: taskId = e1c7ff8e-d980-431e-b745-f13723ef7d75 >2014-09-29 12:17:53,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] spmStart polling ended: taskId = e1c7ff8e-d980-431e-b745-f13723ef7d75 task status = finished >2014-09-29 12:17:53,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] spmStart polling ended, spm status: SPM >2014-09-29 12:17:53,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] START, HSMClearTaskVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, taskId=e1c7ff8e-d980-431e-b745-f13723ef7d75), log id: 35eb65d4 >2014-09-29 12:17:53,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] FINISH, HSMClearTaskVDSCommand, log id: 35eb65d4 >2014-09-29 12:17:53,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11a2412b, log id: 5bc73179 >2014-09-29 12:17:53,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [4270fff5] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-09-29 12:17:53,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4270fff5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05 (Address: monique-vds05.tlv.redhat.com). >2014-09-29 12:17:53,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4270fff5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: cda6349 >2014-09-29 12:17:53,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4270fff5] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-29 12:17:53,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4270fff5] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 32f8dae4 >2014-09-29 12:17:53,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4270fff5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32f8dae4 >2014-09-29 12:17:53,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4270fff5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cda6349 >2014-09-29 12:17:53,753 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-10) [4270fff5] Discovered no tasks on Storage Pool DC1 >2014-09-29 12:17:55,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@12c49f04 >2014-09-29 12:17:55,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:17:55,957 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:55,957 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:17:55,994 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:17:56,111 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) START, SetVmStatusVDSCommand( vmId = 0855adca-3940-4a9d-ac40-63bc25f26c34, status = Unknown, exit status = Normal), log id: 7604ac29 >2014-09-29 12:17:56,137 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, SetVmStatusVDSCommand, log id: 7604ac29 >2014-09-29 12:17:56,324 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM HostedEngine was set to the Unknown status. >2014-09-29 12:17:56,325 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 1, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:56,355 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 12:17:56,358 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-5) ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 12:17:56,373 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:17:56,486 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-5) [fef3fb4] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 12:17:56,489 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-5) [fef3fb4] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-09-29 12:17:56,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@47972c91 >2014-09-29 12:17:56,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:17:56,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:56,975 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:17:57,077 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:17:57,182 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-5) [fef3fb4] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-09-29 12:17:57,192 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:17:57,288 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 12:17:57,290 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-21) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 12:17:57,329 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:17:57,496 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-21) [f392626] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 12:17:57,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-21) [f392626] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 12:17:57,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-21) [f392626] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:17:57,857 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-21) [f392626] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 12:17:57,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-21) [f392626] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: b357251 >2014-09-29 12:17:58,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-21) [f392626] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: b357251 >2014-09-29 12:17:58,478 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-21) [f392626] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:17:58,763 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-21) [f392626] Executing SSH Soft Fencing command on host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:19:56,217 WARN [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp-/127.0.0.1:8702-9) [70801ee7] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,VDS_CANNOT_MAINTENANCE_VDS_IS_NOT_RESPONDING_WITH_VMS,$HostNotResponding monique-vds01.tlv.redhat.com >2014-09-29 12:20:59,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@dead312 >2014-09-29 12:20:59,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:20:59,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:20:59,755 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:20:59,783 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:20:59,783 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:20:59,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-5) [fef3fb4] Correlation ID: fef3fb4, Job ID: 811e3b8f-cba0-4561-b206-39b46a414e83, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 12:21:00,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4ed75476 >2014-09-29 12:21:00,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:21:00,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:21:00,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:21:00,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:21:00,895 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:21:00,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-21) [f392626] Correlation ID: f392626, Job ID: a21c1790-bde4-4a7a-ada6-59af4b720b55, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 12:24:03,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@19e17a26 >2014-09-29 12:24:03,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:24:03,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:24:03,233 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:24:03,271 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:24:03,271 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:24:03,275 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:24:04,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6bac8b00 >2014-09-29 12:24:04,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:24:04,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:24:04,243 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:24:04,350 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:24:04,351 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:24:04,354 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:27:06,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@22e5cabd >2014-09-29 12:27:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:27:06,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:27:06,434 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:27:06,463 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:27:06,467 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) START, SetVmStatusVDSCommand( vmId = 0855adca-3940-4a9d-ac40-63bc25f26c34, status = Unknown, exit status = Normal), log id: 6729e39f >2014-09-29 12:27:06,493 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, SetVmStatusVDSCommand, log id: 6729e39f >2014-09-29 12:27:06,536 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM HostedEngine was set to the Unknown status. >2014-09-29 12:27:06,536 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 1, spm_status = None, non-responsive_timeout (seconds) = 120, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:27:06,609 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 12:27:06,612 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-1) ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 12:27:06,642 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:27:06,673 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-1) [685b3a87] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:27:06,674 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-1) [685b3a87] Failed to run Fence script on vds:monique-vds01.tlv.redhat.com. >2014-09-29 12:27:06,708 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-1) [685b3a87] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-09-29 12:27:06,709 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-1) [685b3a87] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-09-29 12:27:06,709 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-1) [685b3a87] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:27:07,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2272a193 >2014-09-29 12:27:07,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:27:07,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:27:07,436 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:27:07,526 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:27:07,531 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 120, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:27:07,619 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 12:27:07,622 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-10) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 12:27:07,664 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:27:07,790 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-10) [a20041e] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:27:08,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 12:27:08,059 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-10) Using Host monique-vds05 from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:27:08,118 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-10) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 12:27:08,335 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-10) [a20041e] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 12:27:38,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-10) [a20041e] Correlation ID: a20041e, Job ID: 5b74b75b-bb00-49df-94b5-06e481018866, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-09-29 12:27:38,557 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-10) [a20041e] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:27:38,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-10) [a20041e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 12:27:38,890 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-10) [a20041e] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:27:38,933 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-10) [a20041e] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 12:27:38,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-10) [a20041e] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 23f859fa >2014-09-29 12:27:39,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-10) [a20041e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 23f859fa >2014-09-29 12:27:39,671 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:27:39,719 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Running command: StopVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 12:27:39,762 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40738775 >2014-09-29 12:30:09,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1a20d924 >2014-09-29 12:30:09,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:30:09,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:30:09,686 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:30:09,738 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:30:09,785 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:30:11,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@cdef6c8 >2014-09-29 12:30:11,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:30:11,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:30:11,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:30:11,314 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:30:11,385 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:30:11,403 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] FINISH, SetVdsStatusVDSCommand, log id: 40738775 >2014-09-29 12:30:11,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 12:30:11,637 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Using Host monique-vds05 from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 12:30:11,680 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Executing <Stop> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=true, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 12:30:11,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=true, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 3656f361 >2014-09-29 12:30:11,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] FINISH, FenceVdsVDSCommand, return: Test Succeeded, skipped, log id: 3656f361 >2014-09-29 12:30:11,918 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 59cd54d8 >2014-09-29 12:30:11,953 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] FINISH, SetVdsStatusVDSCommand, log id: 59cd54d8 >2014-09-29 12:30:12,033 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com became non responsive and was not restarted due to the Cluster Fencing Policy. >2014-09-29 12:30:12,049 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-10) [701d0cfc] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 12:33:13,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@11840afd >2014-09-29 12:33:13,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:33:13,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:33:13,199 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:33:13,233 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:33:13,281 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:33:14,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@79fb503c >2014-09-29 12:33:14,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:33:14,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:33:14,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:33:14,800 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:33:14,848 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:36:16,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1f5b8012 >2014-09-29 12:36:16,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:36:16,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:36:16,798 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:36:16,824 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:36:16,874 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:36:18,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@490aa8a8 >2014-09-29 12:36:18,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:36:18,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:36:18,356 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:36:18,448 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:36:18,500 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:39:20,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2ed97784 >2014-09-29 12:39:20,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:39:20,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:39:20,011 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:39:20,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:39:20,089 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:39:22,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78487085 >2014-09-29 12:39:22,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:39:22,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:39:22,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:39:22,153 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:39:22,201 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:42:23,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6560bf16 >2014-09-29 12:42:23,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:42:23,177 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:42:23,177 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:42:23,204 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:42:23,240 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:42:25,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@25244112 >2014-09-29 12:42:25,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:42:25,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:42:25,705 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:42:25,795 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:42:25,842 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:45:26,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1835c1f4 >2014-09-29 12:45:26,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:45:26,255 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:45:26,255 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:45:26,282 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:45:26,329 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:45:29,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [462f5c31] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e766a5a >2014-09-29 12:45:29,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [462f5c31] HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:45:29,269 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [462f5c31] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:45:29,269 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [462f5c31] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:45:29,363 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [462f5c31] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:45:29,410 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [462f5c31] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:48:29,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@58aa1255 >2014-09-29 12:48:29,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [29bf91d6] HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:48:29,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:48:29,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:48:29,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:48:29,973 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [29bf91d6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:48:32,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@63b4e297 >2014-09-29 12:48:32,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:48:32,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:48:32,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:48:32,980 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:48:33,021 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:51:33,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f640bdb >2014-09-29 12:51:33,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:51:33,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:51:33,085 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:51:33,145 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:51:33,193 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:51:36,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1a4ababd >2014-09-29 12:51:36,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:51:36,121 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:51:36,121 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:51:36,201 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:51:36,241 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:54:36,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3fa19c45 >2014-09-29 12:54:36,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:54:36,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) [4270fff5] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:54:36,570 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) [4270fff5] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:54:36,587 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4270fff5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:54:36,618 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) [4270fff5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:54:39,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@586b454d >2014-09-29 12:54:39,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:54:39,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:54:39,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:54:39,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:54:39,719 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:57:39,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7c1c914 >2014-09-29 12:57:39,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) HostName = monique-vds01.tlv.redhat.com >2014-09-29 12:57:39,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:57:39,791 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:57:39,818 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:57:39,865 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 12:57:42,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@189aa053 >2014-09-29 12:57:42,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 12:57:42,825 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 12:57:42,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:57:42,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 12:57:43,003 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:00:42,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@58e59b6 >2014-09-29 13:00:42,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:00:42,921 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:00:42,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:00:42,953 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:00:43,055 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [18ad7c77] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:00:46,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38206a46 >2014-09-29 13:00:46,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:00:46,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:00:46,445 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:00:46,538 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:00:46,585 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:02:26,705 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:02:27,009 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:02:27,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 68e74c73 >2014-09-29 13:03:46,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b03e4bd >2014-09-29 13:03:46,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:03:46,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:03:46,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:03:46,517 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:03:46,565 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:03:50,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7025a5d0 >2014-09-29 13:03:50,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:03:50,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:03:50,026 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:03:50,126 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:03:50,174 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:05:05,105 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:05:05,341 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:05:05,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 16d2fc07 >2014-09-29 13:05:13,725 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-9) [39c2eb0b] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:05:13,726 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-26) [39c2eb0b] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:05:13,887 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-7-thread-26) [3b623d11] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:05:14,077 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-26) [3b623d11] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 322ac464 >2014-09-29 13:05:27,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=5022, mMessage=Message timeout which can be caused by communication issues']] >2014-09-29 13:05:27,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:05:27,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Command SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:05:27,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 68e74c73 >2014-09-29 13:05:27,530 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Command org.ovirt.engine.core.bll.MaintenanceVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-29 13:05:27,558 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-41) [7b18443c] Correlation ID: 7b18443c, Job ID: 3f70101f-bc13-4eab-8e8e-cff5b9fe520d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot change into maintenance mode - not all Vms have been migrated successfully. Consider manual intervention: stopping/migrating Vms: <UNKNOWN> (User: admin). >2014-09-29 13:05:54,241 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:05:54,494 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:05:54,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 1d65482d >2014-09-29 13:06:49,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3241be6b >2014-09-29 13:06:49,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:06:49,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:06:49,867 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:06:49,894 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:06:49,937 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:06:53,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@365ada27 >2014-09-29 13:06:53,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:06:53,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:06:53,382 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:06:53,471 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:06:53,519 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:06:53,547 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-26) [3b623d11] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 322ac464 >2014-09-29 13:06:53,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-26) [3b623d11] Correlation ID: 3b623d11, Call Stack: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host rose07.qa.lab.tlv.redhat.com were changed to 'Down' by admin >2014-09-29 13:06:53,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-26) [3b623d11] Correlation ID: 39c2eb0b, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-09-29 13:06:53,694 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-26) [3b623d11] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:07:43,429 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-09-29 13:07:44,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-09-29 13:08:05,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=5022, mMessage=Message timeout which can be caused by communication issues']] >2014-09-29 13:08:05,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:08:05,573 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Command SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:08:05,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 16d2fc07 >2014-09-29 13:08:05,574 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Command org.ovirt.engine.core.bll.MaintenanceVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-29 13:08:05,595 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-6) [41b238f0] Correlation ID: 41b238f0, Job ID: 5aa48a0a-76dd-4cd7-8b69-cc40b480adf1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot change into maintenance mode - not all Vms have been migrated successfully. Consider manual intervention: stopping/migrating Vms: <UNKNOWN> (User: admin). >2014-09-29 13:08:20,871 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 540 seconds while maximum configured value is 300 seconds. >2014-09-29 13:08:55,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=5022, mMessage=Message timeout which can be caused by communication issues']] >2014-09-29 13:08:55,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:08:55,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] Command SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:08:55,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1d65482d >2014-09-29 13:08:55,003 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-18) [695eb06] Command org.ovirt.engine.core.bll.MaintenanceVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-29 13:08:55,040 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-18) [695eb06] Correlation ID: 695eb06, Job ID: e61f1471-f023-45a1-839b-6b0c4fceec4f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot change into maintenance mode - not all Vms have been migrated successfully. Consider manual intervention: stopping/migrating Vms: <UNKNOWN> (User: admin). >2014-09-29 13:09:48,242 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-11) [4fef5034] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:09:48,244 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-4) [4fef5034] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:09:48,362 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-7-thread-4) [66c7c438] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:09:48,486 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-4) [66c7c438] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 616557ad >2014-09-29 13:09:53,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2bec3e8 >2014-09-29 13:09:53,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:09:53,016 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:09:53,016 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:09:53,047 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:09:53,055 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:09:57,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@34b41349 >2014-09-29 13:09:57,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:09:57,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:09:57,054 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:09:57,149 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:09:57,185 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:09:57,212 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-4) [66c7c438] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 616557ad >2014-09-29 13:09:57,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-4) [66c7c438] Correlation ID: 66c7c438, Call Stack: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host rose07.qa.lab.tlv.redhat.com were changed to 'Down' by admin >2014-09-29 13:09:57,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-4) [66c7c438] Correlation ID: 4fef5034, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-09-29 13:09:57,342 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-4) [66c7c438] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:10:04,571 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-7) [f0d4f5c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:10:05,007 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-7) [f0d4f5c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:10:05,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-7) [f0d4f5c] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 425048a5 >2014-09-29 13:10:29,579 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-28) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:10:29,760 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-43) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:10:30,099 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:10:33,112 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:10:34,593 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-67) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:10:34,812 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-52) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:10:36,121 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:11:19,676 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-29 13:11:19,690 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-29 13:11:19,691 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-29 13:11:19,691 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-29 13:11:19,721 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-29 13:11:19,721 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-29 13:11:19,721 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-29 13:11:19,722 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-29 13:11:19,722 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-29 13:11:19,723 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-29 13:11:19,724 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DB_USER" is "engine". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-29 13:11:19,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-29 13:11:19,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-29 13:11:19,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-29 13:11:19,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-09-29 13:11:19,729 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_USER" is "ovirt". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-29 13:11:19,730 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-29 13:11:19,731 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-29 13:11:19,731 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-29 13:11:19,746 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 43) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-29 13:11:38,877 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 53) Start initializing Backend >2014-09-29 13:11:40,036 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 53) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-29 13:11:40,781 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 53) Running ovirt-engine 3.5.0-0.13.beta.el6ev >2014-09-29 13:11:40,781 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 53) Start initializing dictionaries >2014-09-29 13:11:40,786 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 53) Finished initializing dictionaries >2014-09-29 13:11:40,786 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Start initializing AuditLogCleanupManager >2014-09-29 13:11:40,787 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-29 13:11:40,850 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Finished initializing AuditLogCleanupManager >2014-09-29 13:11:40,850 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Start initializing CommandEntityCleanupManager >2014-09-29 13:11:40,851 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-29 13:11:40,853 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 53) Finished initializing CommandEntityCleanupManager >2014-09-29 13:11:40,883 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 53) Start initializing TagsDirector >2014-09-29 13:11:40,912 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 53) Tag root added to tree >2014-09-29 13:11:40,980 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 53) Finished initializing TagsDirector >2014-09-29 13:11:40,981 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 53) Start initializing IsoDomainListSyncronizer >2014-09-29 13:11:40,990 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 53) Finished initializing IsoDomainListSyncronizer >2014-09-29 13:11:41,109 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 53) Completed initializing handlers >2014-09-29 13:11:41,201 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 53) Start initializing ErrorTranslatorImpl >2014-09-29 13:11:41,251 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 53) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-29 13:11:41,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 53) Finished initializing ErrorTranslatorImpl >2014-09-29 13:11:41,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 53) Start initializing ErrorTranslatorImpl >2014-09-29 13:11:41,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 53) Finished initializing ErrorTranslatorImpl >2014-09-29 13:11:41,254 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 53) Mark incomplete jobs as UNKNOWN >2014-09-29 13:11:41,469 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 53) Start initializing JobRepositoryCleanupManager >2014-09-29 13:11:41,470 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 53) Finished initializing JobRepositoryCleanupManager >2014-09-29 13:11:41,471 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 53) Start initializing AutoRecoveryManager >2014-09-29 13:11:41,472 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 53) Finished initializing AutoRecoveryManager >2014-09-29 13:11:41,475 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 53) Start initializing ExecutionMessageDirector >2014-09-29 13:11:41,477 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 53) Finished initializing ExecutionMessageDirector >2014-09-29 13:11:41,771 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Loading extension 'builtin-authn-internal' >2014-09-29 13:11:41,784 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Extension 'builtin-authn-internal' loaded >2014-09-29 13:11:41,787 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Loading extension 'internal' >2014-09-29 13:11:41,790 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Extension 'internal' loaded >2014-09-29 13:11:41,791 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Initializing extension 'builtin-authn-internal' >2014-09-29 13:11:41,791 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Extension 'builtin-authn-internal' initialized >2014-09-29 13:11:41,812 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Initializing extension 'internal' >2014-09-29 13:11:41,813 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Extension 'internal' initialized >2014-09-29 13:11:41,813 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Start of enabled extensions list >2014-09-29 13:11:41,813 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 13:11:41,814 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 13:11:41,814 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 53) End of enabled extensions list >2014-09-29 13:11:42,333 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 53) Start initializing DbUserCacheManager >2014-09-29 13:11:42,349 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 53) Finished initializing DbUserCacheManager >2014-09-29 13:11:42,381 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 53) Initialization of AsyncTaskManager completed successfully. >2014-09-29 13:11:42,384 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 53) Start initializing ResourceManager >2014-09-29 13:11:42,866 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Entered VdsManager constructor >2014-09-29 13:11:42,891 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-09-29 13:11:42,981 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-09-29 13:11:43,535 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 53) VDS f10f0801-7d59-421d-9d4a-a5fa3b874b13 was added to the Resource Manager >2014-09-29 13:11:43,535 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Entered VdsManager constructor >2014-09-29 13:11:43,568 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-09-29 13:11:43,599 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 53) VDS 08c6d6dd-82dd-41e7-9e92-6957151a40e3 was added to the Resource Manager >2014-09-29 13:11:43,600 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Entered VdsManager constructor >2014-09-29 13:11:43,641 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 53) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-09-29 13:11:43,672 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 53) VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 was added to the Resource Manager >2014-09-29 13:11:43,742 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 53) Finished initializing ResourceManager >2014-09-29 13:11:43,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 53) Initialization of OvfDataUpdater completed successfully. >2014-09-29 13:11:43,763 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Start scheduling to enable vds load balancer >2014-09-29 13:11:43,764 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Finished scheduling to enable vds load balancer >2014-09-29 13:11:43,764 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Start HA Reservation check >2014-09-29 13:11:43,764 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Finished HA Reservation check >2014-09-29 13:11:43,798 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 53) Init VM custom properties utilities >2014-09-29 13:11:43,799 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 53) Init device custom properties utilities >2014-09-29 13:11:43,804 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Initializing Scheduling manager >2014-09-29 13:11:43,817 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-09-29 13:11:43,842 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-09-29 13:11:43,962 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) External scheduler disabled, discovery skipped >2014-09-29 13:11:43,962 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 53) Initialized Scheduling manager >2014-09-29 13:11:43,962 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 53) Initializing DWH Heart Beat >2014-09-29 13:11:43,967 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 53) DWH Heart Beat initialized >2014-09-29 13:11:47,118 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-09-29 13:11:47,437 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:11:48,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-29 13:11:48,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-29 13:11:48,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-29 13:11:48,687 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-29 13:11:48,687 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-29 13:11:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-29 13:11:48,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 18b11280 >2014-09-29 13:11:48,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 494d4d47 >2014-09-29 13:11:48,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 494d4d47 >2014-09-29 13:11:48,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 18b11280 >2014-09-29 13:11:50,577 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [6107f674] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:11:50,578 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [271dfdfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:11:50,759 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 539 seconds while maximum configured value is 300 seconds. >2014-09-29 13:11:50,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [271dfdfe] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 4f8cbe21 >2014-09-29 13:11:50,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [271dfdfe] FINISH, GetHardwareInfoVDSCommand, log id: 4f8cbe21 >2014-09-29 13:11:50,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6107f674] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 4aa0b1c6 >2014-09-29 13:11:51,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6107f674] FINISH, GetHardwareInfoVDSCommand, log id: 4aa0b1c6 >2014-09-29 13:11:51,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 2a515579 >2014-09-29 13:11:51,272 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [1db5dbd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:11:51,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 2a515579 >2014-09-29 13:11:51,345 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [41777b37] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:11:51,426 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:11:51,432 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:11:51,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7f077b3a >2014-09-29 13:11:51,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 327f59a1 >2014-09-29 13:11:51,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 7f077b3a >2014-09-29 13:11:51,907 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-09-29 13:11:51,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 327f59a1 >2014-09-29 13:11:51,964 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-29 13:11:52,146 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05 >2014-09-29 13:11:52,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-7) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vdsId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 19c8d463 >2014-09-29 13:11:52,299 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [2cdd4233] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 13:11:52,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 19c8d463 >2014-09-29 13:11:53,829 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [767a353] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:11:53,848 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [767a353] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-29 13:11:54,066 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [767a353] Correlation ID: 767a353, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-09-29 13:11:54,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [767a353] hostFromVds::selectedVds - monique-vds05, spmStatus SPM, storage pool DC1 >2014-09-29 13:11:54,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [767a353] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-09-29 13:11:54,511 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [767a353] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05 (Address: monique-vds05.tlv.redhat.com). >2014-09-29 13:11:54,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [767a353] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: 2b37a3b4 >2014-09-29 13:11:54,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [767a353] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-29 13:11:54,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [767a353] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 4b249012 >2014-09-29 13:11:54,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [767a353] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b249012 >2014-09-29 13:11:54,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [767a353] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b37a3b4 >2014-09-29 13:11:54,922 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-10) [767a353] Discovered no tasks on Storage Pool DC1 >2014-09-29 13:12:12,416 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-29 13:12:12,417 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-29 13:12:12,558 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-29 13:12:12,559 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-29 13:12:17,946 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com has time-drift of 515 seconds while maximum configured value is 300 seconds. >2014-09-29 13:12:18,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 52907616 >2014-09-29 13:12:18,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 52907616 >2014-09-29 13:12:18,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-17) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 6313ff1b >2014-09-29 13:12:18,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-09-29 13:12:18,327 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 13:12:18,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Correlation ID: 41777b37, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 13:12:18,443 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [4fd23fc4] Initializing Host: monique-vds01.tlv.redhat.com >2014-09-29 13:12:18,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-17) FINISH, ConnectStoragePoolVDSCommand, log id: 6313ff1b >2014-09-29 13:12:18,476 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [312372b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:12:19,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3565fc94] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:12:19,211 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [3565fc94] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:12:19,331 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [3565fc94] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:12:19,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: b4340f6 >2014-09-29 13:12:20,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: b4340f6 >2014-09-29 13:12:20,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 18cc2fd7 >2014-09-29 13:12:20,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [3565fc94] FINISH, SetMOMPolicyParametersVDSCommand, log id: 18cc2fd7 >2014-09-29 13:12:20,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3565fc94] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-09-29 13:12:20,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3565fc94] Correlation ID: 1db5dbd, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-09-29 13:12:20,536 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [3565fc94] Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-09-29 13:12:20,648 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [6edcc124] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:12:26,077 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginAdminUserCommand internal: false. >2014-09-29 13:12:26,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-29 13:12:36,763 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-09-29 13:12:36,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-29 13:18:56,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetStatsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:18:56,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:18:56,974 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:18:56,974 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) vds::refreshVdsStats Failed getVdsStats, vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, error = VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:18:56,974 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand.executeVdsBrokerCommand(GetStatsVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:695) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:18:56,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:19:06,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-87) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:19:06,462 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:19:06,559 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:19:06,560 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:19:06,613 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:22:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@56eb64a1 >2014-09-29 13:22:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:22:00,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:22:00,183 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:22:00,223 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:22:00,223 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:22:00,230 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:22:09,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1d85ea8e >2014-09-29 13:22:09,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:22:09,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:22:09,782 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:22:09,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:22:09,888 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:22:09,892 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:25:03,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@ec85f36 >2014-09-29 13:25:03,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:25:03,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:25:03,473 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:25:03,508 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:25:03,616 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:25:03,654 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 13:25:03,681 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-13) ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 13:25:03,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:25:03,909 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-13) [2bc495f] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:25:03,913 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-13) [2bc495f] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-09-29 13:25:04,985 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-13) [2bc495f] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-09-29 13:25:13,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6c80bc7 >2014-09-29 13:25:13,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 13:25:13,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:25:13,063 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:25:13,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:25:13,306 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:25:13,403 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 13:25:13,406 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-25) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 13:25:13,442 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:25:13,599 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:25:14,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:25:14,104 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:25:14,155 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:25:14,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-25) [7a11934b] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 37f20d2e >2014-09-29 13:25:14,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-25) [7a11934b] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 37f20d2e >2014-09-29 13:25:14,764 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:25:14,985 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-25) [7a11934b] Executing SSH Soft Fencing command on host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:26:44,863 ERROR [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (DefaultQuartzScheduler_Worker-17) Error updating DWH Heart Beat: : org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1001) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1064) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:388) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:351) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:136) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:74) [dal.jar:] > at org.ovirt.engine.core.dao.dwh.DwhHistoryTimekeepingDaoDbFacadeImpl.save(DwhHistoryTimekeepingDaoDbFacadeImpl.java:36) [dal.jar:] > at org.ovirt.engine.core.bll.dwh.DwhHeartBeat.engineIsRunningNotification(DwhHeartBeat.java:42) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 16 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 18 more > >2014-09-29 13:26:45,076 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-29) Failed to invoke scheduled method performLoadBalancing: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.scheduling.SchedulingManager.performLoadBalancing(SchedulingManager.java:852) [bll.jar:] > ... 7 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:26:45,442 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-24) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:26:45,488 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-32) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:26:45,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:26:48,706 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:26:50,479 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-39) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:26:50,493 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-36) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-29 13:26:51,710 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:95) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:40) [dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:35) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:220) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 21 more > >2014-09-29 13:27:16,844 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-29 13:27:16,897 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-29 13:27:16,897 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-29 13:27:16,898 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-29 13:27:16,898 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-29 13:27:16,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-29 13:27:16,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-29 13:27:16,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-29 13:27:16,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-29 13:27:16,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-29 13:27:16,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-29 13:27:16,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-29 13:27:16,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DB_USER" is "engine". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-29 13:27:16,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-29 13:27:16,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-29 13:27:16,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-29 13:27:16,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-09-29 13:27:16,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-09-29 13:27:16,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-29 13:27:16,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-29 13:27:16,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-29 13:27:16,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-29 13:27:16,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-29 13:27:16,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_USER" is "ovirt". >2014-09-29 13:27:16,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-29 13:27:16,941 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-29 13:27:16,941 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-29 13:27:16,941 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-29 13:27:16,941 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-29 13:27:16,941 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-29 13:27:35,213 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Start initializing Backend >2014-09-29 13:27:37,787 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 49) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-29 13:27:38,347 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Running ovirt-engine 3.5.0-0.13.beta.el6ev >2014-09-29 13:27:38,347 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 49) Start initializing dictionaries >2014-09-29 13:27:38,351 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 49) Finished initializing dictionaries >2014-09-29 13:27:38,351 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Start initializing AuditLogCleanupManager >2014-09-29 13:27:38,352 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-29 13:27:38,460 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Finished initializing AuditLogCleanupManager >2014-09-29 13:27:38,460 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Start initializing CommandEntityCleanupManager >2014-09-29 13:27:38,460 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-29 13:27:38,462 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Finished initializing CommandEntityCleanupManager >2014-09-29 13:27:38,465 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Start initializing TagsDirector >2014-09-29 13:27:38,537 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Tag root added to tree >2014-09-29 13:27:38,578 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Finished initializing TagsDirector >2014-09-29 13:27:38,579 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 49) Start initializing IsoDomainListSyncronizer >2014-09-29 13:27:38,588 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 49) Finished initializing IsoDomainListSyncronizer >2014-09-29 13:27:38,803 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Completed initializing handlers >2014-09-29 13:27:38,911 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Start initializing ErrorTranslatorImpl >2014-09-29 13:27:38,959 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-29 13:27:38,960 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Finished initializing ErrorTranslatorImpl >2014-09-29 13:27:38,960 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Start initializing ErrorTranslatorImpl >2014-09-29 13:27:38,961 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Finished initializing ErrorTranslatorImpl >2014-09-29 13:27:38,961 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Mark incomplete jobs as UNKNOWN >2014-09-29 13:27:39,088 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 49) Start initializing JobRepositoryCleanupManager >2014-09-29 13:27:39,089 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 49) Finished initializing JobRepositoryCleanupManager >2014-09-29 13:27:39,090 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 49) Start initializing AutoRecoveryManager >2014-09-29 13:27:39,091 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 49) Finished initializing AutoRecoveryManager >2014-09-29 13:27:39,094 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 49) Start initializing ExecutionMessageDirector >2014-09-29 13:27:39,097 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 49) Finished initializing ExecutionMessageDirector >2014-09-29 13:27:39,251 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Loading extension 'builtin-authn-internal' >2014-09-29 13:27:39,268 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'builtin-authn-internal' loaded >2014-09-29 13:27:39,271 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Loading extension 'internal' >2014-09-29 13:27:39,274 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'internal' loaded >2014-09-29 13:27:39,275 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Initializing extension 'builtin-authn-internal' >2014-09-29 13:27:39,275 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'builtin-authn-internal' initialized >2014-09-29 13:27:39,298 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Initializing extension 'internal' >2014-09-29 13:27:39,299 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'internal' initialized >2014-09-29 13:27:39,299 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Start of enabled extensions list >2014-09-29 13:27:39,300 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 13:27:39,300 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-29 13:27:39,300 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) End of enabled extensions list >2014-09-29 13:27:39,779 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 49) Start initializing DbUserCacheManager >2014-09-29 13:27:39,781 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 49) Finished initializing DbUserCacheManager >2014-09-29 13:27:39,823 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 49) Initialization of AsyncTaskManager completed successfully. >2014-09-29 13:27:39,826 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) Start initializing ResourceManager >2014-09-29 13:27:40,339 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Entered VdsManager constructor >2014-09-29 13:27:40,369 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-09-29 13:27:40,454 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-09-29 13:27:41,034 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) VDS 08c6d6dd-82dd-41e7-9e92-6957151a40e3 was added to the Resource Manager >2014-09-29 13:27:41,034 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Entered VdsManager constructor >2014-09-29 13:27:41,066 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-09-29 13:27:41,125 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) VDS f10f0801-7d59-421d-9d4a-a5fa3b874b13 was added to the Resource Manager >2014-09-29 13:27:41,126 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Entered VdsManager constructor >2014-09-29 13:27:41,167 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-09-29 13:27:41,237 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) VDS eec426a9-0846-4c02-83f0-c6fdb9566de0 was added to the Resource Manager >2014-09-29 13:27:41,349 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) Finished initializing ResourceManager >2014-09-29 13:27:41,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 49) Initialization of OvfDataUpdater completed successfully. >2014-09-29 13:27:41,388 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Start scheduling to enable vds load balancer >2014-09-29 13:27:41,389 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Finished scheduling to enable vds load balancer >2014-09-29 13:27:41,389 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Start HA Reservation check >2014-09-29 13:27:41,390 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Finished HA Reservation check >2014-09-29 13:27:41,428 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-09-29 13:27:41,434 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 49) Init VM custom properties utilities >2014-09-29 13:27:41,434 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 49) Init device custom properties utilities >2014-09-29 13:27:41,486 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Initializing Scheduling manager >2014-09-29 13:27:41,491 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-09-29 13:27:41,631 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) External scheduler disabled, discovery skipped >2014-09-29 13:27:41,631 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Initialized Scheduling manager >2014-09-29 13:27:41,631 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 49) Initializing DWH Heart Beat >2014-09-29 13:27:41,632 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 49) DWH Heart Beat initialized >2014-09-29 13:27:42,745 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-2) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-29 13:27:42,804 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-2) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-29 13:27:42,945 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-09-29 13:27:42,946 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-09-29 13:27:44,818 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-09-29 13:27:44,998 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:27:47,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 4f800006 >2014-09-29 13:27:47,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 4f800006 >2014-09-29 13:27:47,958 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-29 13:27:47,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-29 13:27:47,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-29 13:27:47,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-29 13:27:47,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-29 13:27:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-29 13:27:47,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-29 13:27:47,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-29 13:27:48,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-29 13:27:48,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-29 13:27:48,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 2453132a >2014-09-29 13:27:48,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 2453132a >2014-09-29 13:27:49,355 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [33d608ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:27:49,354 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [43ba7b60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:27:49,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [33d608ce] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 1f020d08 >2014-09-29 13:27:49,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [33d608ce] FINISH, GetHardwareInfoVDSCommand, log id: 1f020d08 >2014-09-29 13:27:49,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [43ba7b60] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 4cc894ff >2014-09-29 13:27:49,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [43ba7b60] FINISH, GetHardwareInfoVDSCommand, log id: 4cc894ff >2014-09-29 13:27:50,640 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [2761fd54] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:27:50,673 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [594d8bc4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:27:50,887 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:27:50,882 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:27:51,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 37b0cfb9 >2014-09-29 13:27:51,211 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 has time-drift of 539 seconds while maximum configured value is 300 seconds. >2014-09-29 13:27:51,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42105d53 >2014-09-29 13:27:51,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 37b0cfb9 >2014-09-29 13:27:51,546 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-09-29 13:27:51,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 42105d53 >2014-09-29 13:27:51,619 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-29 13:27:51,621 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:27:51,629 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Storage Pool 43e1738a-3a93-481a-befe-65c12fbc155f - Updating Storage Domain af66b5d4-dd51-491d-b2cb-c88c1ae89d41 status from Active to Unknown, reason : null >2014-09-29 13:27:52,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, vds=Host[monique-vds05,eec426a9-0846-4c02-83f0-c6fdb9566de0]), log id: 698fd5a2 >2014-09-29 13:27:52,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-5) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vdsId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 7eda58f4 >2014-09-29 13:27:52,343 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Correlation ID: 15eaf6f4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-09-29 13:27:52,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 698fd5a2 >2014-09-29 13:27:52,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-5) FINISH, ConnectStoragePoolVDSCommand, log id: 7eda58f4 >2014-09-29 13:27:53,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] hostFromVds::selectedVds - monique-vds05, spmStatus SPM, storage pool DC1 >2014-09-29 13:27:53,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-09-29 13:27:53,468 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05 >2014-09-29 13:27:53,605 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [4b1123df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: eec426a9-0846-4c02-83f0-c6fdb9566de0 Type: VDS >2014-09-29 13:27:53,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05 (Address: monique-vds05.tlv.redhat.com). >2014-09-29 13:27:54,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, ignoreFailoverLimit = false), log id: 2f51a2d2 >2014-09-29 13:27:54,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] -- executeIrsBrokerCommand: Attempting on storage pool 43e1738a-3a93-481a-befe-65c12fbc155f >2014-09-29 13:27:54,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0), log id: 24dfe67a >2014-09-29 13:27:54,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24dfe67a >2014-09-29 13:27:54,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f51a2d2 >2014-09-29 13:27:54,288 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-10) [15eaf6f4] Discovered no tasks on Storage Pool DC1 >2014-09-29 13:27:55,947 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-09-29 13:27:56,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-29 13:27:59,431 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginAdminUserCommand internal: false. >2014-09-29 13:27:59,636 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-09-29 13:28:17,808 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com has time-drift of 515 seconds while maximum configured value is 300 seconds. >2014-09-29 13:28:17,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 582c984 >2014-09-29 13:28:17,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-20) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 363880a7 >2014-09-29 13:28:18,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [729d8be] FINISH, SetMOMPolicyParametersVDSCommand, log id: 582c984 >2014-09-29 13:28:18,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [729d8be] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-09-29 13:28:18,104 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [729d8be] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 13:28:18,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [729d8be] Correlation ID: 2761fd54, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 13:28:18,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 363880a7 >2014-09-29 13:28:18,199 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [729d8be] Initializing Host: monique-vds01.tlv.redhat.com >2014-09-29 13:28:18,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [4676bc36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:28:18,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [7267e06a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:28:18,478 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [7267e06a] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:28:18,561 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [7267e06a] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:28:18,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 8378337 >2014-09-29 13:28:18,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 8378337 >2014-09-29 13:28:18,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 46fb4c24 >2014-09-29 13:28:18,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-7) [7267e06a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 46fb4c24 >2014-09-29 13:28:18,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [7267e06a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-09-29 13:28:18,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [7267e06a] Correlation ID: 594d8bc4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-09-29 13:28:19,048 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [7267e06a] Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-09-29 13:28:19,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:28:48,327 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:28:48,327 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-09-29 13:28:48,375 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:28:48,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1c4c027d >2014-09-29 13:28:48,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1c4c027d >2014-09-29 13:28:50,070 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 13:28:50,070 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 13:28:50,071 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-7) [6618ef52] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-29 13:28:50,638 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 13:28:50,645 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-29 13:28:51,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:28:51,487 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-7) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:28:51,536 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:28:51,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-7) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 139a76b8 >2014-09-29 13:28:51,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 139a76b8 >2014-09-29 13:28:51,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: 6618ef52, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-09-29 13:28:56,106 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-2) [12ff33ec] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:28:56,258 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-2) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-09-29 13:28:56,258 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-2) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-09-29 13:28:56,259 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-2) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:29:21,956 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-1) [5c575101] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:29:22,039 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-1) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-09-29 13:29:22,039 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-1) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-09-29 13:29:22,039 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-1) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:29:59,399 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-6) [2fcb963] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:29:59,483 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-6) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-09-29 13:29:59,483 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-6) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-09-29 13:29:59,483 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-6) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:31:03,793 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-5) [795c3085] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:31:03,833 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-5) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-09-29 13:31:03,833 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-5) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-09-29 13:31:03,833 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-5) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:32:19,241 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-3) [5a53d4ed] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:32:19,290 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-3) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-09-29 13:32:19,290 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-3) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-09-29 13:32:19,290 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-3) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:33:15,320 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-7) [1f1383f6] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:33:15,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:15,597 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:15,656 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] Running command: RestartVdsCommand internal: false. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:33:16,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:16,123 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:16,175 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:33:16,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 588cfa2a >2014-09-29 13:33:16,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [1f1383f6] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 588cfa2a >2014-09-29 13:33:16,883 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Running command: StopVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:33:17,078 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35d1612e >2014-09-29 13:33:17,144 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] FINISH, SetVdsStatusVDSCommand, log id: 35d1612e >2014-09-29 13:33:18,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:18,069 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:18,138 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:33:18,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 18715933 >2014-09-29 13:33:18,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 18715933 >2014-09-29 13:33:18,264 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-09-29 13:33:23,390 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:23,390 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:23,390 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 13:33:23,422 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:33:23,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1f22e0e8 >2014-09-29 13:33:23,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 1f22e0e8 >2014-09-29 13:33:23,526 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] vds rose07.qa.lab.tlv.redhat.com status is off >2014-09-29 13:33:23,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) [ff1a2c2] Correlation ID: ff1a2c2, Job ID: c8d9b694-2b43-4f22-a466-f6860dc1d5e3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-09-29 13:33:23,998 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-31) [5e598023] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:33:24,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) [5e598023] Correlation ID: 5e598023, Job ID: c8d9b694-2b43-4f22-a466-f6860dc1d5e3, Call Stack: null, Custom Event ID: -1, Message: Auto fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-09-29 13:33:24,332 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:24,333 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:24,344 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-31) Running command: StartVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:33:24,380 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-31) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3df918a3 >2014-09-29 13:33:24,395 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-31) FINISH, SetVdsStatusVDSCommand, log id: 3df918a3 >2014-09-29 13:33:24,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:24,575 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:24,610 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:33:24,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 46a01842 >2014-09-29 13:33:24,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 46a01842 >2014-09-29 13:33:24,719 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-31) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-09-29 13:33:27,932 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:33:29,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:33:29,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:33:29,791 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-31) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 13:33:29,824 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:33:29,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 4b0835e0 >2014-09-29 13:34:30,942 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:30,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:30,951 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:30,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:34:30,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:31,108 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:31,108 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:34,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-31) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4b0835e0 >2014-09-29 13:34:34,073 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-31) vds rose07.qa.lab.tlv.redhat.com status is on >2014-09-29 13:34:34,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) Correlation ID: 7ee85af, Job ID: c8d9b694-2b43-4f22-a466-f6860dc1d5e3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-09-29 13:34:34,233 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:34:34,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-31) Correlation ID: 1f1383f6, Job ID: c8d9b694-2b43-4f22-a466-f6860dc1d5e3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-09-29 13:34:34,294 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-7-thread-31) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:34:49,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:49,236 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:49,237 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:49,240 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:34:49,241 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:49,296 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [15eaf6f4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:52,357 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:34:52,358 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:52,359 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:52,359 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:52,363 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:34:52,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:52,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:55,516 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:34:55,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:55,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:55,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:55,525 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:34:55,525 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:55,614 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:58,670 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:34:58,671 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:58,672 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:58,672 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:34:58,675 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:34:58,675 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:34:58,731 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:35:01,778 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:35:08,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:35:08,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:35:08,781 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:35:08,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:35:08,793 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:35:08,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:35:11,911 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:35:12,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 3fb6063e >2014-09-29 13:35:12,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, GetHardwareInfoVDSCommand, log id: 3fb6063e >2014-09-29 13:35:12,700 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [5627ad31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:35:12,723 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [65b1fc1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:35:12,752 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:35:12,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3c6dde6c >2014-09-29 13:35:13,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 3c6dde6c >2014-09-29 13:35:13,074 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-29 13:35:13,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 7ad39f7e >2014-09-29 13:35:13,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 7ad39f7e >2014-09-29 13:35:13,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:35:13,595 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:35:13,636 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:35:13,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 67399b75 >2014-09-29 13:35:13,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 67399b75 >2014-09-29 13:35:13,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 4d3169ff >2014-09-29 13:35:13,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [5a7604f8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d3169ff >2014-09-29 13:35:13,879 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-09-29 13:35:13,906 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [5a7604f8] Correlation ID: 65b1fc1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-09-29 13:35:14,018 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [2499c4d5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:39:37,856 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-9) [6e8e2859] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-09-29 13:39:37,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [6e8e2859] Correlation ID: 6e8e2859, Call Stack: null, Custom Event ID: -1, Message: Event log message 4774 was removed by User admin. >2014-09-29 13:39:39,304 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-2) [ada193f] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-09-29 13:39:39,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [ada193f] Correlation ID: ada193f, Call Stack: null, Custom Event ID: -1, Message: Event log message 4967 was removed by User admin. >2014-09-29 13:39:41,578 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-4) [2234b6ef] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-09-29 13:39:41,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [2234b6ef] Correlation ID: 2234b6ef, Call Stack: null, Custom Event ID: -1, Message: Event log message 4800 was removed by User admin. >2014-09-29 13:39:52,096 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [6da65430] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: 0795bb40-4675-4601-a800-70a45c3ddf02 Type: VdsGroupsAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN >2014-09-29 13:39:52,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [6da65430] Correlation ID: 6da65430, Call Stack: null, Custom Event ID: -1, Message: Host cluster cluster1 was updated by admin >2014-09-29 13:40:35,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-81) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:40:35,803 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:40:35,975 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:40:35,976 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:40:36,023 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:40:39,842 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:41:42,842 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:41:42,843 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:41:42,844 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:41:42,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:41:42,847 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:41:42,892 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:41:42,897 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection timed out >2014-09-29 13:41:45,935 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:42:48,935 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:42:48,936 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:42:48,936 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:42:48,940 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:42:48,940 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:42:49,050 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.ConnectException: Connection timed out >2014-09-29 13:42:49,106 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 13:42:49,133 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-30) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 13:42:49,195 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:42:49,360 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-30) [c454cae] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:42:49,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [c454cae] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:42:49,608 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [c454cae] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:42:49,663 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [c454cae] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:42:49,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [c454cae] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 6df897ec >2014-09-29 13:42:49,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [c454cae] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6df897ec >2014-09-29 13:42:49,794 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-30) [c454cae] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:42:52,241 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:43:52,893 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-30) [c454cae] SSH Soft Fencing command failed on host rose07.qa.lab.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view7.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$4.run(VdsEventListener.java:249) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-09-29 13:43:52,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [c454cae] Correlation ID: c454cae, Job ID: 789fcea5-6e12-4e81-8a56-f08c1ce54d99, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 13:43:53,006 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:43:53,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:43:53,114 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:43:53,123 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-30) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:43:53,215 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:43:55,241 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:43:55,242 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:43:55,243 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:43:55,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:43:55,246 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:43:55,297 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:43:55,333 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:43:58,375 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:44:23,338 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [38a09232] Correlation ID: 38a09232, Job ID: e9759544-d2e5-4def-b132-a6c53de5a8f8, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-09-29 13:44:23,343 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:44:23,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [38a09232] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:44:23,469 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [38a09232] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:44:23,501 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [38a09232] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:44:23,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 5550ddf8 >2014-09-29 13:45:01,375 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:45:01,376 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:45:01,377 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:45:01,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:45:01,380 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:45:01,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:45:04,470 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:45:54,153 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-80) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:45:54,154 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:45:54,172 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:45:54,172 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:45:54,181 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:46:07,470 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:46:07,471 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:46:07,472 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:46:07,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:46:07,480 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:46:07,594 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:46:11,137 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:47:14,137 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:47:14,138 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:47:14,138 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:47:14,146 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:47:14,147 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:47:14,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:47:14,198 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:47:17,240 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:48:20,240 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:48:20,241 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:48:20,242 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:48:20,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:48:20,245 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:48:20,332 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:48:23,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:48:57,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [4676bc36] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1f8f0556 >2014-09-29 13:48:57,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [4676bc36] HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:48:57,275 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [4676bc36] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:48:57,275 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [4676bc36] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:48:57,283 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [4676bc36] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:48:57,283 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [4676bc36] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:48:57,295 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [4676bc36] Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:49:26,404 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:49:26,405 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:49:26,406 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:49:26,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:49:26,414 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:49:26,454 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:49:29,536 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:50:17,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] FINISH, FenceVdsVDSCommand, return: Test Failed, Message timeout which can be caused by communication issues', log id: 5550ddf8 >2014-09-29 13:50:17,316 WARN [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [38a09232] Fencing operation failed with proxy host f10f0801-7d59-421d-9d4a-a5fa3b874b13, trying another proxy... >2014-09-29 13:50:17,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [38a09232] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:17,407 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [38a09232] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:17,407 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [38a09232] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:50:17,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 479d28ef >2014-09-29 13:50:17,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [38a09232] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 479d28ef >2014-09-29 13:50:18,039 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:50:18,048 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Running command: StopVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:50:18,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5baca5a3 >2014-09-29 13:50:32,536 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:50:32,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:50:32,537 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:50:32,540 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:50:32,540 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:50:32,619 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:50:32,664 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:50:32,693 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] FINISH, SetVdsStatusVDSCommand, log id: 5baca5a3 >2014-09-29 13:50:32,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:32,835 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Using Host monique-vds05 from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:32,870 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Executing <Stop> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 13:50:32,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 72b05fe4 >2014-09-29 13:50:33,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 72b05fe4 >2014-09-29 13:50:33,035 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-09-29 13:50:38,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:38,180 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:38,180 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 13:50:38,212 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 13:50:38,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 47df1803 >2014-09-29 13:50:38,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 47df1803 >2014-09-29 13:50:38,336 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] vds rose07.qa.lab.tlv.redhat.com status is off >2014-09-29 13:50:38,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [1ab3c757] Correlation ID: 1ab3c757, Job ID: 15139539-8e33-44f9-9c81-322736778ac3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-09-29 13:50:38,440 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-30) [17439e57] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:50:38,454 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [17439e57] Correlation ID: 17439e57, Job ID: cb00abca-8612-4b44-806e-e991a9f0ba81, Call Stack: null, Custom Event ID: -1, Message: Auto fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-09-29 13:50:38,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:38,642 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:38,648 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-30) Running command: StartVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 13:50:38,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 71bd3c15 >2014-09-29 13:50:38,685 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) FINISH, SetVdsStatusVDSCommand, log id: 71bd3c15 >2014-09-29 13:50:38,756 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:50:38,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:38,823 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Using Host monique-vds05 from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:38,854 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Executing <Start> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 13:50:38,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 28ecf376 >2014-09-29 13:50:38,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 28ecf376 >2014-09-29 13:50:38,962 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-30) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-09-29 13:50:44,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:50:44,032 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:50:44,032 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-30) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 13:50:44,064 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-30) Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 13:50:44,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 4ca5a29a >2014-09-29 13:50:44,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-30) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4ca5a29a >2014-09-29 13:50:44,168 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-30) vds rose07.qa.lab.tlv.redhat.com status is on >2014-09-29 13:50:44,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: 7c8d0750, Job ID: cff198d2-bca9-4fa1-ad71-dcc82142a516, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-09-29 13:50:44,198 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) Correlation ID: 38a09232, Job ID: 4c8c4234-5017-468c-acdc-506faa92e46b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-09-29 13:50:44,210 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-30) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 13:51:41,756 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:51:41,757 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:51:41,758 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:51:41,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-29 13:51:41,762 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:51:41,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:51:44,971 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:52:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17e2d43c >2014-09-29 13:52:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [14cc77bc] HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:52:00,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:52:00,478 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:52:00,509 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:52:00,699 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:52:00,706 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 13:52:00,709 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-50) [14cc77bc] ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 13:52:00,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [14cc77bc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:52:00,763 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-50) [530704ed] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:52:00,766 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-50) [530704ed] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-09-29 13:52:01,437 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-50) [530704ed] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-09-29 13:52:15,972 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:52:15,972 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:52:15,975 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.7.0_51] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.7.0_51] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:104) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.process(SSLEngineNioHelper.java:125) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.pendingOperations(SSLClient.java:74) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:52:15,976 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:101) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:58) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:52:15,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-29 13:52:15,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:52:16,072 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_51] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:84) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-29 13:52:19,112 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-09-29 13:52:26,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]), log id: 13b7d592 >2014-09-29 13:52:26,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, GetHardwareInfoVDSCommand, log id: 13b7d592 >2014-09-29 13:52:27,024 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [570ec1f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:52:27,039 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [20c92a93] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:52:27,046 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:52:27,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36c69fb7 >2014-09-29 13:52:27,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 36c69fb7 >2014-09-29 13:52:27,320 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-09-29 13:52:27,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-33) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 4a4e0509 >2014-09-29 13:52:27,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 4a4e0509 >2014-09-29 13:52:27,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 13:52:27,714 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 13:52:27,746 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 13:52:27,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 76ea01a7 >2014-09-29 13:52:27,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 76ea01a7 >2014-09-29 13:52:27,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3), log id: 6cbdc40b >2014-09-29 13:52:27,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-31) [3a70c8f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6cbdc40b >2014-09-29 13:52:28,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-09-29 13:52:28,036 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [3a70c8f] Correlation ID: 20c92a93, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-09-29 13:52:28,120 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [4f082233] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 13:55:04,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2a4e2c2e >2014-09-29 13:55:04,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) HostName = monique-vds01.tlv.redhat.com >2014-09-29 13:55:04,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 13:55:04,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:55:04,058 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 13:55:04,059 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 13:55:04,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-50) [530704ed] Correlation ID: 530704ed, Job ID: da3eb463-4022-498d-9952-c8bc9bc9a09d, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 13:55:07,070 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-09-29 13:55:07,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]), log id: 8d29c5e >2014-09-29 13:55:07,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GetHardwareInfoVDSCommand, log id: 8d29c5e >2014-09-29 13:55:07,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [609cf0ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 13:55:07,759 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [5650e92f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:55:07,769 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 43e1738a-3a93-481a-befe-65c12fbc155f Type: StoragePool >2014-09-29 13:55:07,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, storageType = NFS, connectionList = [{ id: a2842de2-9bf4-4a96-ae54-fa0a6f633b3a, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_rhevm_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cb6be38 >2014-09-29 13:55:07,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] FINISH, ConnectStorageServerVDSCommand, return: {a2842de2-9bf4-4a96-ae54-fa0a6f633b3a=0}, log id: 6cb6be38 >2014-09-29 13:55:07,809 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-09-29 13:55:07,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vdsId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, storagePoolId = 43e1738a-3a93-481a-befe-65c12fbc155f, masterVersion = 1), log id: 1dfabcb9 >2014-09-29 13:55:08,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 1dfabcb9 >2014-09-29 13:55:33,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13), log id: 2bb2a43a >2014-09-29 13:55:33,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [5871981a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2bb2a43a >2014-09-29 13:55:33,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [5871981a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-09-29 13:55:33,520 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [5871981a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-09-29 13:55:33,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [5871981a] Correlation ID: 5650e92f, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 13:55:33,549 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [7f05b475] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 14:01:29,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetStatsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:01:29,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:01:29,383 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:01:29,383 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) vds::refreshVdsStats Failed getVdsStats, vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, error = VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:01:29,383 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand.executeVdsBrokerCommand(GetStatsVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:695) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:01:29,410 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:01:49,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-20) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:01:49,495 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:01:49,586 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:01:49,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:997) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:946) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:664) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:01:49,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:04:32,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@11c9be98 >2014-09-29 14:04:32,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:04:32,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:04:32,748 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:04:32,775 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:04:32,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:04:32,779 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:04:52,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e440aca >2014-09-29 14:04:52,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:04:52,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:04:52,929 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:04:53,022 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:04:53,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:04:53,026 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:07:35,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5b71aeb7 >2014-09-29 14:07:35,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:07:35,896 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:07:35,896 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:07:35,920 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:07:35,925 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:07:35,951 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 14:07:35,953 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-33) ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 14:07:35,976 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:07:36,107 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-33) [74af4be8] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: f10f0801-7d59-421d-9d4a-a5fa3b874b13 Type: VDS >2014-09-29 14:07:36,113 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-33) [74af4be8] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-09-29 14:07:36,621 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-33) [74af4be8] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-09-29 14:07:56,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@63f97357 >2014-09-29 14:07:56,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:07:56,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:07:56,550 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:07:56,651 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:07:56,755 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:07:56,844 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 14:07:56,847 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-7) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 14:07:56,863 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:07:57,002 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-7) [486bf47] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 14:07:57,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-7) [486bf47] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:07:57,283 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-7) [486bf47] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:07:57,327 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-7) [486bf47] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 14:07:57,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-7) [486bf47] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 283b83b1 >2014-09-29 14:07:57,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-7) [486bf47] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 283b83b1 >2014-09-29 14:07:57,946 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-7) [486bf47] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:07:58,257 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-7-thread-7) [486bf47] Executing SSH Soft Fencing command on host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:10:39,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@73855eec >2014-09-29 14:10:39,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:10:39,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:10:39,431 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:10:39,459 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:10:39,459 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:10:39,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [74af4be8] Correlation ID: 74af4be8, Job ID: 1b241671-89c3-48eb-a7e1-0976664b8a98, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 14:11:00,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1fff745a >2014-09-29 14:11:00,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:11:00,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:11:00,044 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:11:00,147 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:11:00,147 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:11:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-7) [486bf47] Correlation ID: 486bf47, Job ID: 1fa6da25-0f78-4db3-abbe-80988a22d8f4, Call Stack: null, Custom Event ID: -1, Message: Unrecognized audit log type has been used. >2014-09-29 14:13:42,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1f602699 >2014-09-29 14:13:42,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:13:42,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:13:42,884 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:13:42,945 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:13:42,945 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:13:42,949 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Failed to refresh VDS , vds = f10f0801-7d59-421d-9d4a-a5fa3b874b13 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:14:03,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@486660e3 >2014-09-29 14:14:03,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:14:03,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:14:03,599 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:14:03,678 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:14:03,678 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:14:03,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 08c6d6dd-82dd-41e7-9e92-6957151a40e3 : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:16:46,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@276bcf7 >2014-09-29 14:16:46,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:16:46,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:16:46,044 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:16:46,072 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:16:46,077 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Server failed to respond, vds_id = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 120, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:16:46,104 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-09-29 14:16:46,107 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-22) ResourceManager::vdsNotResponding entered for Host f10f0801-7d59-421d-9d4a-a5fa3b874b13, monique-vds01.tlv.redhat.com >2014-09-29 14:16:46,130 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:16:46,149 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-22) [318ce5b] Lock Acquired to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 14:16:46,149 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-22) [318ce5b] Failed to run Fence script on vds:monique-vds01.tlv.redhat.com. >2014-09-29 14:16:46,155 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-22) [318ce5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-09-29 14:16:46,155 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-22) [318ce5b] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-09-29 14:16:46,155 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-22) [318ce5b] Lock freed to object EngineLock [exclusiveLocks= key: f10f0801-7d59-421d-9d4a-a5fa3b874b13 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 14:17:07,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6dde6afc >2014-09-29 14:17:07,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:17:07,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:17:07,226 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:17:07,320 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:17:07,426 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Server failed to respond, vds_id = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 120, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:17:07,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-09-29 14:17:07,521 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-7-thread-43) ResourceManager::vdsNotResponding entered for Host 08c6d6dd-82dd-41e7-9e92-6957151a40e3, rose07.qa.lab.tlv.redhat.com >2014-09-29 14:17:07,574 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:17:07,637 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 14:17:07,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:17:07,888 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Using Host monique-vds05 from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:17:07,936 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-43) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 14:17:08,177 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDS >2014-09-29 14:17:38,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Correlation ID: 3856e78e, Job ID: 3953887a-39b3-45c1-ac84-2b9fa0138fc9, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-09-29 14:17:38,336 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 14:17:38,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:17:38,546 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:17:38,588 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [3856e78e] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-09-29 14:17:38,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3856e78e] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1d6532de >2014-09-29 14:17:39,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [3856e78e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1d6532de >2014-09-29 14:17:39,366 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Lock Acquired to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ] >2014-09-29 14:17:39,491 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Running command: StopVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 14:17:39,534 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40491435 >2014-09-29 14:19:49,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3d722357 >2014-09-29 14:19:49,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = monique-vds01.tlv.redhat.com >2014-09-29 14:19:49,668 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = f10f0801-7d59-421d-9d4a-a5fa3b874b13, vds=Host[monique-vds01.tlv.redhat.com,f10f0801-7d59-421d-9d4a-a5fa3b874b13]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:19:49,668 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:19:49,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:19:49,745 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:20:10,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5a610fd7 >2014-09-29 14:20:10,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) HostName = rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:10,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, vds=Host[rose07.qa.lab.tlv.redhat.com,08c6d6dd-82dd-41e7-9e92-6957151a40e3]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-29 14:20:10,869 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:20:10,959 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-29 14:20:11,007 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:654) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:501) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-29 14:20:11,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] FINISH, SetVdsStatusVDSCommand, log id: 40491435 >2014-09-29 14:20:11,244 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:20:11,244 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Using Host monique-vds05 from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:11,287 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Executing <Stop> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 14:20:11,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 2d9ee35 >2014-09-29 14:20:11,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 2d9ee35 >2014-09-29 14:20:11,461 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-09-29 14:20:16,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:20:16,618 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:16,618 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 14:20:16,660 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 14:20:16,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: b75b826 >2014-09-29 14:20:16,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: b75b826 >2014-09-29 14:20:16,796 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] vds rose07.qa.lab.tlv.redhat.com status is off >2014-09-29 14:20:16,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [1cdd1e9f] Correlation ID: 1cdd1e9f, Job ID: 2bfea420-7b65-4155-bb28-7cc29929651a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-09-29 14:20:17,130 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-43) [6267a7ed] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 14:20:17,159 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) [6267a7ed] Correlation ID: 6267a7ed, Job ID: b761ba52-c3d0-4e28-83ce-d2056a016048, Call Stack: null, Custom Event ID: -1, Message: Auto fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-09-29 14:20:17,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:20:17,557 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:17,608 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-43) Running command: StartVdsCommand internal: true. Entities affected : ID: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-29 14:20:17,651 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 23f44e37 >2014-09-29 14:20:17,689 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, SetVdsStatusVDSCommand, log id: 23f44e37 >2014-09-29 14:20:17,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:20:17,897 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Using Host monique-vds05 from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:17,941 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Executing <Start> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 14:20:17,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 311bd774 >2014-09-29 14:20:18,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 311bd774 >2014-09-29 14:20:18,087 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-43) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-09-29 14:20:23,253 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05 from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-09-29 14:20:23,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Using Host monique-vds05 from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-09-29 14:20:23,253 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-43) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-09-29 14:20:23,296 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-43) Executing <Status> Power Management command, Proxy Host:monique-vds05, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-09-29 14:20:23,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, FenceVdsVDSCommand(HostName = monique-vds05, HostId = eec426a9-0846-4c02-83f0-c6fdb9566de0, targetVdsId = 08c6d6dd-82dd-41e7-9e92-6957151a40e3, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 788cc324 >2014-09-29 14:20:23,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 788cc324 >2014-09-29 14:20:23,423 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-43) vds rose07.qa.lab.tlv.redhat.com status is on >2014-09-29 14:20:23,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: 2ce7d47d, Job ID: d0fdb643-fe3b-4d13-ac83-6f30ceb0c516, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-09-29 14:20:23,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-43) Correlation ID: 3856e78e, Job ID: de35f021-1435-4675-aca1-c4575d010237, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-09-29 14:20:23,523 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-7-thread-43) Lock freed to object EngineLock [exclusiveLocks= key: 08c6d6dd-82dd-41e7-9e92-6957151a40e3 value: VDS_FENCE >, sharedLocks= ]
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1147487
: 942291