Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 930541 Details for
Bug 1133611
No health check alert is issued in ui after first message was issued
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
engine_25_08_14.log (text/x-log), 4.56 MB, created by
sefi litmanovich
on 2014-08-25 14:45:53 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
sefi litmanovich
Created:
2014-08-25 14:45:53 UTC
Size:
4.56 MB
patch
obsolete
>2014-08-18 03:28:43,711 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) [17a6802f] Principal admin::internal synchronized >2014-08-18 03:28:46,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-08-18 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) Start deleteAgedOutAuditLogs >2014-08-18 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Start deleteAgedOutCommandEntities >2014-08-18 03:35:35,020 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Finished deleteAgedOutCommandEntities >2014-08-18 03:35:35,029 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) Finished deleteAgedOutAuditLogs >2014-08-18 04:28:43,716 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-08-18 04:28:46,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-18 05:28:43,728 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Principal admin::internal synchronized >2014-08-18 05:28:46,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-18 06:28:43,738 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) Principal admin::internal synchronized >2014-08-18 06:28:46,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [14e8b62f] Attempting to update VMs/Templates Ovf. >2014-08-18 07:28:43,743 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-08-18 07:28:46,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-08-18 08:28:43,749 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-08-18 08:28:46,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-08-18 09:28:43,756 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-08-18 09:28:46,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-18 10:28:43,767 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-08-18 10:28:46,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) [43c167db] Attempting to update VMs/Templates Ovf. >2014-08-18 11:15:09,914 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-08-18 11:15:09,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:15:10,014 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 11:15:50,121 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-6) ResourceManager::searchBusinessObjects - erroneous search text - Events:from 14441 >2014-08-18 11:19:15,958 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-18 11:19:15,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:19:16,021 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [1703f750] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:19:16,025 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [542c3aee] Running command: LogoutUserCommand internal: false. >2014-08-18 11:19:16,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [542c3aee] Correlation ID: 542c3aee, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:19:23,605 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 11:19:23,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:19:24,099 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,100 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,101 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_update_77ec04a0-e638-4cad] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:24,122 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,123 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:24,448 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:24,480 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:24,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 32fcc401 >2014-08-18 11:19:25,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 32fcc401 >2014-08-18 11:19:25,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: hosts_update_77ec04a0-e638-4cad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:19:29,272 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_1429d14e-d9f5-40e3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:19:29,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,388 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:29,394 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:29,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,500 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:29,534 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:29,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2ecab895 >2014-08-18 11:19:29,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2ecab895 >2014-08-18 11:19:29,842 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:29,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 63947ecf >2014-08-18 11:19:29,881 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, SetVdsStatusVDSCommand, log id: 63947ecf >2014-08-18 11:19:29,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,985 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:30,051 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:30,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7e658a1f >2014-08-18 11:19:30,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, FenceVdsVDSCommand, return: Done, log id: 7e658a1f >2014-08-18 11:19:30,258 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 11:19:35,260 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:19:35,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:35,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 64d51855 >2014-08-18 11:19:35,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 64d51855 >2014-08-18 11:19:35,614 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 11:19:35,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: 6c0c45ba, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 11:19:35,674 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-3) [7002e688] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:35,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,825 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,830 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:35,862 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 73c6b6b5 >2014-08-18 11:19:35,867 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, SetVdsStatusVDSCommand, log id: 73c6b6b5 >2014-08-18 11:19:35,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,967 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,999 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:36,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4e6a6317 >2014-08-18 11:19:36,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Done, log id: 4e6a6317 >2014-08-18 11:19:36,249 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 11:19:41,250 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:19:41,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:41,324 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:41,356 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:41,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 24d716a7 >2014-08-18 11:19:41,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 24d716a7 >2014-08-18 11:19:41,632 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 11:19:41,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: 5d19c336, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 11:19:41,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_syncAction_1429d14e-d9f5-40e3, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 11:19:41,647 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:21:18,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-18 11:21:18,830 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 11:21:18,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:21:22,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 46379c02 >2014-08-18 11:21:22,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 46379c02 >2014-08-18 11:21:22,741 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [36bc2c13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:21:22,783 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [1d7cb8ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:21:22,809 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:21:22,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40f54bf7 >2014-08-18 11:21:23,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 40f54bf7 >2014-08-18 11:21:23,139 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 11:21:23,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 5d088cad >2014-08-18 11:21:23,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5d088cad >2014-08-18 11:21:23,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:21:23,668 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:21:23,700 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:21:23,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 68ca7585 >2014-08-18 11:21:24,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 68ca7585 >2014-08-18 11:21:24,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 293e7d1f >2014-08-18 11:21:24,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, SetMOMPolicyParametersVDSCommand, log id: 293e7d1f >2014-08-18 11:21:24,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 11:21:24,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: 1d7cb8ec, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 11:21:24,641 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [2b16911e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:10,877 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,878 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,879 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [hosts_update_af885f2a-0c7f-4b17] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:10,887 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,888 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,959 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:10,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: hosts_update_af885f2a-0c7f-4b17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:27:32,013 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:32,017 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:32,043 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [17313e32] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:27:32,055 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [461d86e8] Running command: LogoutUserCommand internal: false. >2014-08-18 11:27:32,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [461d86e8] Correlation ID: 461d86e8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:27:48,742 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:48,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:48,761 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [47c80510] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:27:48,775 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [58ad6aba] Running command: LogoutUserCommand internal: false. >2014-08-18 11:27:48,793 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [58ad6aba] Correlation ID: 58ad6aba, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:27:53,973 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:53,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:54,346 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,347 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,348 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_26bdde25-1df1-4e3d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:54,366 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,367 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:54,590 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:54,622 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:54,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 69527d96 >2014-08-18 11:27:55,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 69527d96 >2014-08-18 11:27:55,338 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_update_26bdde25-1df1-4e3d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:27:59,024 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_6b3017e1-b5f7-4542] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:27:59,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,137 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,150 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:59,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,257 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:59,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 1b89f1ce >2014-08-18 11:27:59,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1b89f1ce >2014-08-18 11:27:59,601 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:59,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6995f951 >2014-08-18 11:27:59,640 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, SetVdsStatusVDSCommand, log id: 6995f951 >2014-08-18 11:27:59,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:59,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4cc9a80f >2014-08-18 11:27:59,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4cc9a80f >2014-08-18 11:28:00,025 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 11:28:05,027 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:28:05,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,100 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,132 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:05,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6f2fe87d >2014-08-18 11:28:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6f2fe87d >2014-08-18 11:28:05,377 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 11:28:05,385 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: 33fa7b6e, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 11:28:05,438 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-4) [c9aa456] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:28:05,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,613 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,619 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:28:05,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 53252745 >2014-08-18 11:28:05,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: 53252745 >2014-08-18 11:28:05,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,767 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,798 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:05,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3a2656ec >2014-08-18 11:28:05,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 3a2656ec >2014-08-18 11:28:06,007 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 11:28:11,008 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:28:11,079 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:11,081 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:11,115 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:11,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2784b300 >2014-08-18 11:28:11,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2784b300 >2014-08-18 11:28:11,366 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 11:28:11,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 27c919c9, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 11:28:11,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_6b3017e1-b5f7-4542, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 11:28:11,400 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:28:43,772 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-08-18 11:28:46,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-08-18 11:29:50,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-18 11:29:50,732 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 11:29:50,734 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:29:54,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 74535bf8 >2014-08-18 11:29:54,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GetHardwareInfoVDSCommand, log id: 74535bf8 >2014-08-18 11:29:54,541 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [747789c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:29:54,551 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [5034b2b4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:29:54,559 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:29:54,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d7aa421 >2014-08-18 11:29:54,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5d7aa421 >2014-08-18 11:29:54,902 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 11:29:54,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 597e00f7 >2014-08-18 11:29:55,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 597e00f7 >2014-08-18 11:29:55,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:29:55,267 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) [476c5856] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:29:55,301 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) [476c5856] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:29:55,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 598dfd59 >2014-08-18 11:29:55,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 598dfd59 >2014-08-18 11:29:55,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: aed5789 >2014-08-18 11:29:55,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, SetMOMPolicyParametersVDSCommand, log id: aed5789 >2014-08-18 11:29:55,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 11:29:55,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: 5034b2b4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 11:29:55,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [6e4aebb0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:36:17,051 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-5) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:39:12,641 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 11:39:12,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:39:12,649 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-3) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host rose07.qa.lab.tlv.redhat.com was restarted by admin >2014-08-18 11:39:12,661 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [330d397e] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:39:12,667 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7ee93c7e] Running command: LogoutUserCommand internal: false. >2014-08-18 11:39:12,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7ee93c7e] Correlation ID: 7ee93c7e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:39:20,200 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 11:39:20,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:39:22,580 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [f57d599] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:39:22,596 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [6d4cdfa8] Running command: LogoutUserCommand internal: false. >2014-08-18 11:39:22,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [6d4cdfa8] Correlation ID: 6d4cdfa8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:42:59,835 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,836 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,840 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_update_3403bf08-d2d3-4160] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:42:59,893 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,894 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,951 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:42:59,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: hosts_update_3403bf08-d2d3-4160, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:44:00,203 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 11:44:00,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:44:01,889 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [4510748b] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:44:01,893 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [727e3475] Running command: LogoutUserCommand internal: false. >2014-08-18 11:44:01,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [727e3475] Correlation ID: 727e3475, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 12:28:43,780 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-08-18 12:28:46,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-18 13:28:43,788 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-18 13:28:46,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-08-18 13:48:06,120 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 13:48:06,134 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:48:06,136 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-4) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:48:06,168 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-4) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 13:48:06,181 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [648e7fab] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:48:06,184 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [681e9e0c] Running command: LogoutUserCommand internal: false. >2014-08-18 13:48:06,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [681e9e0c] Correlation ID: 681e9e0c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:33,496 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:33,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:33,605 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [59af7713] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:33,608 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4a292ec6] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:33,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a292ec6] Correlation ID: 4a292ec6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:33,674 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:33,707 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [2a32b614] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:33,710 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [4428ee7b] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:33,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [4428ee7b] Correlation ID: 4428ee7b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:50,745 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:50,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:50,761 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-1) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:54:50,812 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host monique-vds01.tlv.redhat.com from cluster >2014-08-18 13:54:50,830 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [275898fe] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:50,833 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [4a6dd6da] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:50,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4a6dd6da] Correlation ID: 4a6dd6da, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:59,234 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:59,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:59,248 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-11) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:54:59,279 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-11) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:54:59,283 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [2e3d38bb] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:59,287 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [ad65065] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:59,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [ad65065] Correlation ID: ad65065, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:55:07,814 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 13:55:07,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:55:07,819 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-9) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:55:07,852 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-9) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:55:07,856 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [45c2cc02] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:55:07,860 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [74d766b5] Running command: LogoutUserCommand internal: false. >2014-08-18 13:55:07,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [74d766b5] Correlation ID: 74d766b5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:55:16,506 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 13:55:16,512 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:55:16,513 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-3) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:55:16,553 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-3) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:55:16,557 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [5bd24a23] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:55:16,561 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [50e560ed] Running command: LogoutUserCommand internal: false. >2014-08-18 13:55:16,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [50e560ed] Correlation ID: 50e560ed, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:07:52,847 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:07:52,858 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:07:54,855 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [39c28a66] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:07:54,858 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [3f9efa9b] Running command: LogoutUserCommand internal: false. >2014-08-18 14:07:54,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3f9efa9b] Correlation ID: 3f9efa9b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:09:50,570 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:09:50,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:09:50,581 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-5) ResourceManager::searchBusinessObjects - erroneous search text - Events : sortby time 2 >2014-08-18 14:09:50,585 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [56be8e88] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:09:50,588 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [609465ad] Running command: LogoutUserCommand internal: false. >2014-08-18 14:09:50,596 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [609465ad] Correlation ID: 609465ad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:09:59,178 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:09:59,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:10:00,589 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [85567d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:10:00,595 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [79403ce8] Running command: LogoutUserCommand internal: false. >2014-08-18 14:10:00,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [79403ce8] Correlation ID: 79403ce8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:10:00,734 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 14:10:01,888 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [5d5af326] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:10:01,901 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [5fbe8d31] Running command: LogoutUserCommand internal: false. >2014-08-18 14:10:01,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5fbe8d31] Correlation ID: 5fbe8d31, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:12:31,943 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 14:12:31,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:12:31,963 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [478f3bc] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:12:31,966 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [821cbc4] Running command: LogoutUserCommand internal: false. >2014-08-18 14:12:31,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [821cbc4] Correlation ID: 821cbc4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:13:09,149 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:13:09,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:13:09,208 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [44be12d5] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:13:09,211 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [4e36eebd] Running command: LogoutUserCommand internal: false. >2014-08-18 14:13:09,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4e36eebd] Correlation ID: 4e36eebd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:13:47,511 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:13:47,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:13:47,573 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [2e6154c] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:13:47,576 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [44910958] Running command: LogoutUserCommand internal: false. >2014-08-18 14:13:47,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [44910958] Correlation ID: 44910958, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:23:07,889 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:23:07,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:23:07,927 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6551ab61] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:23:07,930 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [6da1a222] Running command: LogoutUserCommand internal: false. >2014-08-18 14:23:07,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [6da1a222] Correlation ID: 6da1a222, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:24:18,067 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:18,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:19,684 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [6c4485d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:24:19,687 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [859192e] Running command: LogoutUserCommand internal: false. >2014-08-18 14:24:19,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [859192e] Correlation ID: 859192e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:24:28,902 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:28,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:48,757 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:48,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:48,837 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [12c87c9a] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:24:48,840 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [1a2ac492] Running command: LogoutUserCommand internal: false. >2014-08-18 14:24:48,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [1a2ac492] Correlation ID: 1a2ac492, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:25:13,919 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-18 14:25:13,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:25:13,970 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [4dfcd826] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:25:13,984 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [2da4b11a] Running command: LogoutUserCommand internal: false. >2014-08-18 14:25:14,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [2da4b11a] Correlation ID: 2da4b11a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:25:28,920 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 14:25:28,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:26:07,524 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 14:26:07,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:26:07,529 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 14:26:07,587 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [272a238a] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:26:07,590 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [5ace3574] Running command: LogoutUserCommand internal: false. >2014-08-18 14:26:07,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [5ace3574] Correlation ID: 5ace3574, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:26:10,710 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 14:26:10,712 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-9) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 14:26:10,806 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2fc9a2d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:26:10,810 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [64e72e3e] Running command: LogoutUserCommand internal: false. >2014-08-18 14:26:10,814 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [64e72e3e] Correlation ID: 64e72e3e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:28:43,795 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-08-18 14:28:46,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-18 15:00:21,258 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 15:00:21,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:00:21,399 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [70b60610] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:00:21,402 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [148f6aad] Running command: LogoutUserCommand internal: false. >2014-08-18 15:00:21,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [148f6aad] Correlation ID: 148f6aad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:01:56,228 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 15:01:56,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:01:56,497 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [63f795c4] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:01:56,511 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [378107a0] Running command: LogoutUserCommand internal: false. >2014-08-18 15:01:56,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [378107a0] Correlation ID: 378107a0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:08:14,837 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 15:08:14,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:08:14,849 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: ""2795"" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:430) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 15:08:14,962 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [6a8fc4] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:08:14,965 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [5c8108bd] Running command: LogoutUserCommand internal: false. >2014-08-18 15:08:14,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [5c8108bd] Correlation ID: 5c8108bd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:08:26,397 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 15:08:26,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:08:26,413 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "kaki" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:441) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 15:08:26,573 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [2fbe75db] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:08:26,590 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [51b8f278] Running command: LogoutUserCommand internal: false. >2014-08-18 15:08:26,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [51b8f278] Correlation ID: 51b8f278, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:28:43,803 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-18 15:28:46,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Attempting to update VMs/Templates Ovf. >2014-08-18 16:13:26,269 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 16:13:26,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 16:13:26,291 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1f320185] Running command: LogoutBySessionCommand internal: false. >2014-08-18 16:13:26,298 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3619c10e] Running command: LogoutUserCommand internal: false. >2014-08-18 16:13:26,301 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3619c10e] Correlation ID: 3619c10e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 16:13:34,405 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 16:13:34,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 16:13:34,721 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,722 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,723 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_66641311-7d6a-4efb] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:13:34,800 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,801 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:35,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:13:35,486 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:13:35,562 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:13:35,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6c98c2c9 >2014-08-18 16:14:15,865 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:15,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 6c98c2c9 >2014-08-18 16:14:15,869 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:14:16,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:14:16,145 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:14:16,146 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:14:16,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7f1e559e >2014-08-18 16:14:56,432 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:56,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 7f1e559e >2014-08-18 16:14:56,507 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:56,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_66641311-7d6a-4efb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:15:00,573 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_syncAction_a128507f-5f61-4ba3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:15:00,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:00,776 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:00,783 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:15:00,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:00,976 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:01,038 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:15:01,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 58f82e59 >2014-08-18 16:15:41,343 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:15:41,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 58f82e59 >2014-08-18 16:15:41,347 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:15:41,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:41,934 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:41,935 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:15:41,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 680c20af >2014-08-18 16:16:22,185 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:16:22,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 680c20af >2014-08-18 16:16:22,249 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:16:22,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 68588e4b >2014-08-18 16:16:22,477 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:16:22,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 68588e4b >2014-08-18 16:16:22,481 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 4290a94f-9e6e-4eed-a517-3401f613ebdf, trying another proxy... >2014-08-18 16:16:22,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:22,710 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:22,711 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:16:22,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 57f693c5 >2014-08-18 16:16:23,027 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:16:23,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 57f693c5 >2014-08-18 16:16:23,103 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:16:23,165 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 27f8a4f2 >2014-08-18 16:16:23,170 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, SetVdsStatusVDSCommand, log id: 27f8a4f2 >2014-08-18 16:16:23,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:23,361 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:23,426 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:16:23,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 87bf54f >2014-08-18 16:16:43,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Done, log id: 87bf54f >2014-08-18 16:16:43,675 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:16:48,677 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:16:48,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:48,821 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:48,888 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:16:48,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: a28473a >2014-08-18 16:17:29,173 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:17:29,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: a28473a >2014-08-18 16:17:29,177 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:17:29,481 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:17:29,483 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:17:29,484 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:17:29,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1c4e65dd >2014-08-18 16:18:09,730 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:18:09,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 1c4e65dd >2014-08-18 16:18:09,738 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Host rose07.qa.lab.tlv.redhat.com Primary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-18 16:18:09,818 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-18 16:18:09,822 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-18 16:18:10,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:10,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:10,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:10,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 4bec6c3e >2014-08-18 16:18:10,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4bec6c3e >2014-08-18 16:18:10,527 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:18:15,528 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:18:15,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:15,758 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:15,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:15,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 5c6f8788 >2014-08-18 16:18:16,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:18:16,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 5c6f8788 >2014-08-18 16:18:16,164 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:18:16,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:16,499 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:16,501 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:16,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 57e5e1da >2014-08-18 16:18:16,809 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:18:16,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 57e5e1da >2014-08-18 16:18:16,812 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Host rose07.qa.lab.tlv.redhat.com Secondary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-18 16:18:16,941 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-18 16:18:16,943 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-18 16:18:17,006 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to power fence host rose07.qa.lab.tlv.redhat.com. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >2014-08-18 16:18:17,014 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: 697e2f3e, Job ID: 8ea519be-710a-4a69-ba77-a20292347efa, Call Stack: null, Custom Event ID: -1, Message: Failed to stop Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-18 16:18:17,021 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Restart host action failed, updating host a8fdd580-c752-4537-9cae-e49e468db7da to NonResponsive >2014-08-18 16:18:17,089 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 25e8c28d >2014-08-18 16:18:17,095 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, SetVdsStatusVDSCommand, log id: 25e8c28d >2014-08-18 16:18:17,103 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: hosts_syncAction_a128507f-5f61-4ba3, Job ID: 8ea519be-710a-4a69-ba77-a20292347efa, Call Stack: null, Custom Event ID: -1, Message: Failed to restart Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-18 16:18:17,108 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:18:17,109 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [] >2014-08-18 16:18:19,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6b1a9fe0 >2014-08-18 16:18:19,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, GetHardwareInfoVDSCommand, log id: 6b1a9fe0 >2014-08-18 16:18:19,560 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [5560aa4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:18:19,572 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-4) [2797c163] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:18:19,580 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:18:19,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13fc0608 >2014-08-18 16:18:19,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 13fc0608 >2014-08-18 16:18:19,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 16:18:19,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 15a5f06e >2014-08-18 16:18:19,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 15a5f06e >2014-08-18 16:18:20,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:20,165 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:20,227 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:18:20,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7baafda5 >2014-08-18 16:19:00,513 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:00,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 7baafda5 >2014-08-18 16:19:00,516 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:00,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:00,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:00,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:19:00,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 36de443b >2014-08-18 16:19:41,043 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:41,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 36de443b >2014-08-18 16:19:41,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 42fc0b0e >2014-08-18 16:19:41,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, SetMOMPolicyParametersVDSCommand, log id: 42fc0b0e >2014-08-18 16:19:41,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 16:19:41,306 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:41,311 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: 2797c163, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com power management. >2014-08-18 16:19:41,384 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [21cb1557] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,144 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,146 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,148 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_update_b0005a08-f1fa-4eb7] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,155 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,156 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,234 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:50,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_update_b0005a08-f1fa-4eb7, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:50,416 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,417 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,418 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [hosts_update_15d66ad0-bf95-44a1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,430 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,431 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:50,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: hosts_update_15d66ad0-bf95-44a1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:50,816 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,817 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,818 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_7f90fdfa-3ced-4a96] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,902 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:51,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:51,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:51,437 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:51,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 566c6100 >2014-08-18 16:19:52,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 566c6100 >2014-08-18 16:19:52,266 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:52,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1bad30ac >2014-08-18 16:19:52,589 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:52,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1bad30ac >2014-08-18 16:19:52,592 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:52,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:52,917 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:52,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:52,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 66fe1988 >2014-08-18 16:19:53,148 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:53,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 66fe1988 >2014-08-18 16:19:53,226 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:53,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_7f90fdfa-3ced-4a96, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:56,323 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_8a31314d-362c-485e] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:19:56,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:56,527 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:56,533 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:56,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:56,894 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:57,030 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:57,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7718fd3f >2014-08-18 16:19:57,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7718fd3f >2014-08-18 16:19:57,403 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:57,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 48144f0d >2014-08-18 16:19:57,735 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:57,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 48144f0d >2014-08-18 16:19:57,739 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:58,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:58,105 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:58,106 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:58,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 25fadbc2 >2014-08-18 16:19:58,331 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:58,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 25fadbc2 >2014-08-18 16:19:58,486 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:58,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2156f0e7 >2014-08-18 16:19:58,560 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, SetVdsStatusVDSCommand, log id: 2156f0e7 >2014-08-18 16:19:58,773 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:58,776 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:58,992 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:58,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45b3771 >2014-08-18 16:19:59,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, FenceVdsVDSCommand, return: Done, log id: 45b3771 >2014-08-18 16:19:59,298 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:20:04,300 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:20:04,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:04,427 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:04,491 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:04,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6f6a4cb1 >2014-08-18 16:20:04,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6f6a4cb1 >2014-08-18 16:20:04,738 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 16:20:04,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: 3ff5a450, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 16:20:04,833 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-4) [4b874bd2] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:20:05,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:05,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:05,125 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:20:05,187 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: db897b8 >2014-08-18 16:20:05,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: db897b8 >2014-08-18 16:20:05,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:05,383 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:05,448 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:05,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5b1d30ca >2014-08-18 16:20:05,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 5b1d30ca >2014-08-18 16:20:05,752 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 16:20:08,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:08,692 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:08,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:10,754 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:20:10,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:10,886 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:10,950 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:10,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1c4fcafe >2014-08-18 16:20:11,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1c4fcafe >2014-08-18 16:20:11,206 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 16:20:11,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 557029d9, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 16:20:11,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_8a31314d-362c-485e, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 16:20:11,223 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:20:13,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:13,794 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:13,795 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:18,904 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:18,906 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:18,907 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:24,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [79101bc3] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:24,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [79101bc3] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:24,021 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [79101bc3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:29,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:29,122 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:29,124 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:34,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:34,229 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:34,230 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:39,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:39,330 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:39,331 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:44,433 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:44,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:44,437 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:49,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:49,551 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:49,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:54,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:54,653 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:54,655 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:59,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [79101bc3] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:59,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [79101bc3] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:59,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [79101bc3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:04,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:04,879 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:04,898 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:10,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:10,034 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:10,041 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:15,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:15,140 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:15,164 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:20,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:20,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:20,289 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:25,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:25,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:25,398 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:30,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:30,498 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:30,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:35,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:35,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:35,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:39,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6cd4a494 >2014-08-18 16:21:39,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 6cd4a494 >2014-08-18 16:21:39,380 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [b7d0b91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:21:39,391 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [7ed347a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:21:39,399 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:21:39,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1ce0c747 >2014-08-18 16:21:39,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 1ce0c747 >2014-08-18 16:21:39,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 16:21:39,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 74808d86 >2014-08-18 16:21:39,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, ConnectStoragePoolVDSCommand, log id: 74808d86 >2014-08-18 16:21:40,157 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:21:40,159 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:21:40,221 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:21:40,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 688577da >2014-08-18 16:21:40,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 688577da >2014-08-18 16:21:40,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 55e7884 >2014-08-18 16:21:40,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, SetMOMPolicyParametersVDSCommand, log id: 55e7884 >2014-08-18 16:21:40,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 16:21:40,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: 7ed347a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 16:21:40,832 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [24033251] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,556 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,557 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,559 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_30e2bd93-e6b3-449d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,579 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,580 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:23:12,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_30e2bd93-e6b3-449d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:23:12,819 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,820 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,821 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-11) [hosts_update_05738963-a7ff-41b8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,858 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,859 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,911 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:23:12,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: hosts_update_05738963-a7ff-41b8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:28:43,809 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-08-18 16:28:46,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-08-18 17:28:43,818 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-08-18 17:28:46,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-08-18 18:28:43,828 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) [79101bc3] Principal admin::internal synchronized >2014-08-18 18:28:46,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-08-18 19:28:43,835 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-18 19:28:46,430 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-18 20:28:43,842 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-08-18 20:28:46,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-08-18 21:28:43,853 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-08-18 21:28:46,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-08-18 22:28:43,863 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-08-18 22:28:46,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-08-18 23:28:43,868 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-18 23:28:46,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-08-19 00:28:43,879 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) [43c167db] Principal admin::internal synchronized >2014-08-19 00:28:46,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) [6e4aebb0] Attempting to update VMs/Templates Ovf. >2014-08-19 01:28:43,884 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-08-19 01:28:46,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Attempting to update VMs/Templates Ovf. >2014-08-19 02:28:43,890 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-08-19 02:28:46,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [5b09d125] Attempting to update VMs/Templates Ovf. >2014-08-19 03:28:43,894 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized >2014-08-19 03:28:46,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-08-19 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Start deleteAgedOutCommandEntities >2014-08-19 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Start deleteAgedOutAuditLogs >2014-08-19 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Finished deleteAgedOutAuditLogs >2014-08-19 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Finished deleteAgedOutCommandEntities >2014-08-19 04:28:43,906 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Principal admin::internal synchronized >2014-08-19 04:28:46,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-19 05:28:43,920 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) [14e8b62f] Principal admin::internal synchronized >2014-08-19 05:28:46,662 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-08-19 06:28:43,940 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-19 06:28:46,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [2086a3b3] Attempting to update VMs/Templates Ovf. >2014-08-19 07:28:43,950 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-19 07:28:46,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) [24033251] Attempting to update VMs/Templates Ovf. >2014-08-19 08:28:43,955 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-08-19 08:28:46,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-08-19 09:28:43,960 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-19 09:28:46,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-08-19 10:18:07,078 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:28:43,967 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-19 10:28:46,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-08-19 10:30:52,102 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginAdminUserCommand internal: false. >2014-08-19 10:30:52,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:30:52,201 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:14,066 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:14,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:33:14,109 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2c8106d3] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:33:14,115 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [34ddc70e] Running command: LogoutUserCommand internal: false. >2014-08-19 10:33:14,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [34ddc70e] Correlation ID: 34ddc70e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:33:27,794 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:27,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:33:27,852 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [52790aa] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:33:27,856 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1b306f84] Running command: LogoutUserCommand internal: false. >2014-08-19 10:33:27,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1b306f84] Correlation ID: 1b306f84, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:34:20,602 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-19 10:34:20,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:34:24,996 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [2ed4b4cc] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:34:25,003 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [51f1b52b] Running command: LogoutUserCommand internal: false. >2014-08-19 10:34:25,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [51f1b52b] Correlation ID: 51f1b52b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:34:40,151 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-19 10:34:40,187 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:34:40,248 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,249 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,250 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [53259bd7] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:34:40,256 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,257 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,321 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:34:40,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: 53259bd7, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:34:40,348 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [1e15cdc7] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:34:40,351 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [2ad755fa] Running command: LogoutUserCommand internal: false. >2014-08-19 10:34:40,361 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2ad755fa] Correlation ID: 2ad755fa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:45:10,322 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-5) [3bb05623] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-19 10:45:10,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [3bb05623] Correlation ID: 3bb05623, Call Stack: null, Custom Event ID: -1, Message: Data Center DC2, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-08-19 10:45:32,449 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-9) [4b65fcb9] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-19 10:45:32,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [4b65fcb9] Correlation ID: 4b65fcb9, Call Stack: null, Custom Event ID: -1, Message: Cluster Cluster_21 was added by admin >2014-08-19 10:45:49,145 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [42547f73] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: POOL >] >2014-08-19 10:45:49,203 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:49,208 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 57002d35 >2014-08-19 10:45:49,210 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] VDS monique-vds05.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-19 10:45:49,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, ignoreStopFailed = false), log id: 2a83f716 >2014-08-19 10:45:49,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, SpmStopVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 24f14f1e >2014-08-19 10:45:49,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] SpmStopVDSCommand::Stopping SPM on vds monique-vds05.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:45:49,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, SpmStopVDSCommand, log id: 24f14f1e >2014-08-19 10:45:49,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, ResetIrsVDSCommand, log id: 2a83f716 >2014-08-19 10:45:49,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, SetVdsStatusVDSCommand, log id: 57002d35 >2014-08-19 10:45:49,481 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: POOL >] >2014-08-19 10:45:49,537 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:49,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [42547f73] Correlation ID: 42547f73, Job ID: 3a00219a-580c-41b8-add3-4df882d3afa9, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-19 10:45:51,324 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com >2014-08-19 10:45:51,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: monique-vds05.tlv.redhat.com. >2014-08-19 10:45:51,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Removing vds [4290a94f-9e6e-4eed-a517-3401f613ebdf] from the domain in maintenance cache >2014-08-19 10:45:51,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Removing host(s) [4290a94f-9e6e-4eed-a517-3401f613ebdf] from hosts unseen domain report cache >2014-08-19 10:45:51,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 3), log id: 11437dcd >2014-08-19 10:45:56,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, DisconnectStoragePoolVDSCommand, log id: 11437dcd >2014-08-19 10:45:56,415 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:45:56,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 507ef248 >2014-08-19 10:45:56,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 507ef248 >2014-08-19 10:45:57,076 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:45:57,079 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 10:45:57,087 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Correlation ID: 1e0d0acf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 10:45:57,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 10:45:57,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-19 10:45:57,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d5dcc45 >2014-08-19 10:45:57,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling started: taskId = 32c652db-c90e-4c68-8fd4-f20b116e2939 >2014-08-19 10:45:58,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling ended: taskId = 32c652db-c90e-4c68-8fd4-f20b116e2939 task status = finished >2014-08-19 10:45:58,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling ended, spm status: SPM >2014-08-19 10:45:58,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=32c652db-c90e-4c68-8fd4-f20b116e2939), log id: 64c66230 >2014-08-19 10:45:58,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] FINISH, HSMClearTaskVDSCommand, log id: 64c66230 >2014-08-19 10:45:58,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dca6086, log id: 4d5dcc45 >2014-08-19 10:45:58,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-19 10:45:58,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-19 10:45:58,321 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-19) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:45:58,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 7b750113 >2014-08-19 10:45:58,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:45:58,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 31a3ddc5 >2014-08-19 10:45:58,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31a3ddc5 >2014-08-19 10:45:58,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7b750113 >2014-08-19 10:45:58,558 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] Discovered no tasks on Storage Pool DC1 >2014-08-19 10:45:59,275 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS, ID: 682324a4-a272-4d9d-a816-ef3a35d4270f Type: VdsGroups >2014-08-19 10:45:59,288 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] START, RemoveVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 2e022485 >2014-08-19 10:45:59,289 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] vdsManager::disposing >2014-08-19 10:45:59,291 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] FINISH, RemoveVdsVDSCommand, log id: 2e022485 >2014-08-19 10:45:59,295 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] START, AddVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 70d53665 >2014-08-19 10:45:59,296 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] AddVds - entered , starting logic to add VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf >2014-08-19 10:45:59,298 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] AddVds - VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added, will try to add it to the resource manager >2014-08-19 10:45:59,299 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] Entered VdsManager constructor >2014-08-19 10:45:59,312 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-19 10:45:59,313 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-11) [53ddd85b] VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-19 10:45:59,314 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] FINISH, AddVdsVDSCommand, log id: 70d53665 >2014-08-19 10:45:59,343 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-11) [43d7cdf3] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,373 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-11) [43d7cdf3] Lock freed to object EngineLock [exclusiveLocks= key: 5d41097a-d767-4fa2-baf4-eedd5526f920 value: REGISTER_VDS >, sharedLocks= ] >2014-08-19 10:45:59,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [43d7cdf3] Correlation ID: 53ddd85b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:45:59,529 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,647 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,648 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] No changes were detected in setup networks for host monique-vds05.tlv.redhat.com (ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf) >2014-08-19 10:45:59,650 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,650 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,651 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [1f0718dd] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,664 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,665 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1f0718dd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-19 10:45:59,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1f0718dd] Correlation ID: 1f0718dd, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:46:02,321 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-19 10:46:03,953 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8) [509fbafa] Lock Acquired to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS >, sharedLocks= ] >2014-08-19 10:46:03,986 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:46:03,988 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Before acquiring lock in order to prevent monitoring for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:03,989 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Lock acquired, from now a monitoring of host will be skipped for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:04,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 72da14bf >2014-08-19 10:46:04,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] FINISH, SetVdsStatusVDSCommand, log id: 72da14bf >2014-08-19 10:46:04,055 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Activate finished. Lock released. Monitoring can run now for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:04,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Correlation ID: 509fbafa, Job ID: 74ba3d5a-9eb3-49f4-9dd5-1b5d143d0b02, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was activated by admin. >2014-08-19 10:46:04,065 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Lock freed to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS >, sharedLocks= ] >2014-08-19 10:46:05,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 57ca2011 >2014-08-19 10:46:05,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 57ca2011 >2014-08-19 10:46:05,776 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:46:05,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 60976a90 >2014-08-19 10:46:05,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] FINISH, GetHardwareInfoVDSCommand, log id: 60976a90 >2014-08-19 10:46:05,967 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-19 10:46:05,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 3c9deae3 >2014-08-19 10:46:06,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3c9deae3 >2014-08-19 10:46:06,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [22f00d97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-08-19 10:46:06,112 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [22f00d97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-19 10:46:06,145 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [615e8cb8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:47:54,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:47:54,988 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-19 10:47:54,991 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:47:54,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 14ff0746 >2014-08-19 10:47:55,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 14ff0746 >2014-08-19 10:48:01,113 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,114 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,116 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-12) [593c4211] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:48:01,135 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,136 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:01,371 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:48:01,403 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:01,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1ff0b870 >2014-08-19 10:48:01,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1ff0b870 >2014-08-19 10:48:01,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: 593c4211, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:48:20,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:20,741 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 10:48:20,775 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:20,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4bcc4373 >2014-08-19 10:48:21,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4bcc4373 >2014-08-19 10:48:28,882 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,883 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,884 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [22c94d72] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:48:28,902 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:29,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:29,215 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:48:29,248 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:29,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 368f3820 >2014-08-19 10:48:29,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 368f3820 >2014-08-19 10:48:29,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: 22c94d72, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:52:38,039 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-12) [4b34bd4c] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-19 10:52:38,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:38,182 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:38,202 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:38,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:38,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:38,444 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:38,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 64a23697 >2014-08-19 10:52:39,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 64a23697 >2014-08-19 10:52:39,341 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:39,386 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 34dc00da >2014-08-19 10:52:39,387 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-19 10:52:39,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 1282d046 >2014-08-19 10:52:39,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 68927cce >2014-08-19 10:52:39,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:52:39,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, SpmStopVDSCommand, log id: 68927cce >2014-08-19 10:52:39,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, ResetIrsVDSCommand, log id: 1282d046 >2014-08-19 10:52:39,535 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, SetVdsStatusVDSCommand, log id: 34dc00da >2014-08-19 10:52:39,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:39,683 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:39,715 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:39,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 51a3396b >2014-08-19 10:52:39,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, FenceVdsVDSCommand, return: Done, log id: 51a3396b >2014-08-19 10:52:39,999 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-19 10:52:41,843 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:52:41,851 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 10:52:41,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Correlation ID: 1354ecb5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 10:52:42,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 10:52:42,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-19 10:52:42,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e2db11f >2014-08-19 10:52:42,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling started: taskId = 45e76b22-aeca-404a-8b68-12c36537d155 >2014-08-19 10:52:43,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling ended: taskId = 45e76b22-aeca-404a-8b68-12c36537d155 task status = finished >2014-08-19 10:52:43,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling ended, spm status: SPM >2014-08-19 10:52:43,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=45e76b22-aeca-404a-8b68-12c36537d155), log id: 8779f00 >2014-08-19 10:52:43,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] FINISH, HSMClearTaskVDSCommand, log id: 8779f00 >2014-08-19 10:52:43,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36b4d6dc, log id: 3e2db11f >2014-08-19 10:52:43,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-19 10:52:43,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-19 10:52:43,224 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-45) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:52:43,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 5e56d350 >2014-08-19 10:52:43,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:52:43,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 786eee8b >2014-08-19 10:52:43,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 786eee8b >2014-08-19 10:52:43,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e56d350 >2014-08-19 10:52:43,385 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] Discovered no tasks on Storage Pool DC1 >2014-08-19 10:52:45,000 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-19 10:52:45,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,075 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,113 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:45,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6af5134d >2014-08-19 10:52:45,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6af5134d >2014-08-19 10:52:45,415 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-19 10:52:45,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: 6f6bace1, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-19 10:52:45,501 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-46) [663ff72b] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:45,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,788 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,794 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:45,826 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 541ed097 >2014-08-19 10:52:45,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SetVdsStatusVDSCommand, log id: 541ed097 >2014-08-19 10:52:45,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,942 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,974 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:45,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5e005195 >2014-08-19 10:52:46,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, FenceVdsVDSCommand, return: Done, log id: 5e005195 >2014-08-19 10:52:46,325 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-19 10:52:51,327 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-19 10:52:51,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:51,421 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:51,456 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:51,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 668d50b6 >2014-08-19 10:52:51,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 668d50b6 >2014-08-19 10:52:51,724 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-19 10:52:51,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: 4bc73fe3, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-19 10:52:51,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: 4b34bd4c, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-19 10:52:51,744 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-19 10:54:31,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-19 10:54:31,080 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 10:54:31,081 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 10:54:34,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 504c4e84 >2014-08-19 10:54:35,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, GetHardwareInfoVDSCommand, log id: 504c4e84 >2014-08-19 10:54:35,070 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [28f19b5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:54:35,081 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-76) [6c272a9b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:54:35,089 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:54:35,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c0f9072 >2014-08-19 10:54:35,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4c0f9072 >2014-08-19 10:54:35,364 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-19 10:54:35,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 11db6525 >2014-08-19 10:54:35,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 11db6525 >2014-08-19 10:54:35,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:54:35,701 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) [6898462d] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:54:35,749 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) [6898462d] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:54:35,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3dfa9058 >2014-08-19 10:54:36,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3dfa9058 >2014-08-19 10:54:36,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 69ced075 >2014-08-19 10:54:36,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 69ced075 >2014-08-19 10:54:36,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-19 10:54:36,152 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: 6c272a9b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-19 10:54:36,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [3e33609] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:59:02,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:59:02,363 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 10:59:02,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:59:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 445f6092 >2014-08-19 10:59:03,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 445f6092 >2014-08-19 10:59:08,781 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,783 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,784 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [4b38753f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:59:08,878 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,879 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:09,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:59:09,407 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:59:09,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:59:09,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 763c71a2 >2014-08-19 10:59:09,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 763c71a2 >2014-08-19 10:59:09,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 4b38753f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 11:06:33,112 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-19 11:06:33,114 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-19 11:06:33,114 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-19 11:06:33,115 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-19 11:06:33,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-19 11:06:33,178 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-19 11:06:33,180 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-19 11:06:33,181 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-19 11:06:33,181 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-19 11:06:33,182 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-19 11:06:33,183 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-19 11:06:33,184 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-19 11:06:33,185 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-19 11:06:33,186 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-19 11:06:33,187 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-19 11:06:33,187 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-19 11:06:33,188 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-19 11:06:33,201 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-19 11:06:33,202 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-19 11:06:33,203 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-19 11:06:33,204 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-19 11:06:33,205 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-19 11:06:33,206 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-19 11:06:33,206 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-19 11:06:33,207 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-19 11:06:33,208 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-19 11:06:33,209 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-19 11:06:33,210 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-19 11:06:33,211 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-19 11:06:33,214 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-19 11:06:33,215 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-19 11:06:33,216 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-19 11:06:33,216 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-19 11:06:33,217 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-19 11:06:33,218 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-19 11:06:33,219 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-19 11:06:33,219 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-19 11:06:33,220 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-19 11:06:33,221 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-19 11:06:33,222 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-19 11:06:33,223 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-19 11:06:33,223 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-19 11:06:33,224 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-19 11:06:33,246 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-19 11:06:33,247 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-19 11:06:33,248 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-19 11:06:33,249 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-19 11:06:33,250 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-19 11:06:33,250 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-19 11:06:33,251 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-19 11:06:33,252 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-19 11:06:33,253 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-19 11:06:33,254 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-19 11:06:33,254 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-19 11:06:33,255 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-19 11:06:33,256 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-19 11:06:33,269 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-19 11:06:33,270 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-19 11:06:33,270 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-19 11:06:33,271 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-19 11:06:33,272 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-19 11:06:33,273 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-19 11:06:33,273 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-19 11:06:33,274 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-19 11:06:33,275 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-19 11:06:33,275 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-19 11:06:33,276 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-19 11:06:33,277 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-19 11:06:35,852 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-19 11:06:37,268 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-19 11:06:37,269 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-19 11:06:37,603 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-19 11:06:37,604 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-19 11:06:37,609 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-19 11:06:37,610 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-19 11:06:37,611 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-19 11:06:37,694 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-19 11:06:37,695 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-19 11:06:37,696 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-19 11:06:37,699 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-19 11:06:37,702 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-19 11:06:37,728 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-19 11:06:37,773 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-19 11:06:37,774 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-19 11:06:37,800 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-19 11:06:38,076 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-19 11:06:38,163 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-19 11:06:38,206 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-19 11:06:38,209 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-19 11:06:38,209 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-19 11:06:38,210 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-19 11:06:38,211 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-19 11:06:38,356 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-19 11:06:38,358 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-19 11:06:38,359 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-19 11:06:38,361 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-19 11:06:38,364 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-19 11:06:38,378 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-19 11:06:38,539 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-19 11:06:38,540 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-19 11:06:38,542 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-19 11:06:38,543 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-19 11:06:39,008 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-19 11:06:39,010 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-19 11:06:39,040 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-19 11:06:39,043 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-19 11:06:39,371 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:39,468 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-19 11:06:39,653 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-19 11:06:39,979 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-19 11:06:39,980 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:40,013 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-19 11:06:40,052 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-19 11:06:40,053 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:40,083 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-19 11:06:40,109 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-19 11:06:40,171 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-19 11:06:40,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-19 11:06:40,193 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-19 11:06:40,194 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-19 11:06:40,194 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-19 11:06:40,195 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-19 11:06:40,238 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-19 11:06:40,243 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-19 11:06:40,245 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-19 11:06:40,260 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-19 11:06:40,296 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-19 11:06:40,373 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-19 11:06:40,374 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-19 11:06:40,375 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-19 11:06:40,377 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-19 11:06:45,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 4a2fc64b >2014-08-19 11:06:45,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 5c48835c >2014-08-19 11:06:45,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 40404cd5 >2014-08-19 11:06:45,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 5c48835c >2014-08-19 11:06:45,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 4a2fc64b >2014-08-19 11:06:45,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 40404cd5 >2014-08-19 11:06:46,889 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-19 11:06:46,885 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-19 11:06:46,943 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-19 11:06:47,486 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [434e9f98] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-19 11:06:47,482 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [2962f4d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:06:47,486 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 11:06:50,465 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [1307f0db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 11:06:50,499 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [1307f0db] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 11:06:50,599 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-19 11:06:50,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: UNASSIGNED not exist in string table >2014-08-19 11:06:50,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-19 11:06:50,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-19 11:06:50,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-19 11:06:50,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-19 11:06:50,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-19 11:06:50,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-19 11:06:50,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-19 11:06:50,650 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-19 11:06:50,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-19 11:06:50,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_STOPPED not exist in string table >2014-08-19 11:06:50,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_STARTED not exist in string table >2014-08-19 11:06:50,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_ERROR not exist in string table >2014-08-19 11:06:50,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-19 11:06:50,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-19 11:06:50,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-19 11:06:50,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-19 11:06:51,992 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Correlation ID: 1307f0db, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 11:06:52,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1307f0db] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-19 11:06:52,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1307f0db] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-19 11:06:52,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-19 11:06:52,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 49a9e9c3 >2014-08-19 11:06:53,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 11:06:53,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 7651d1d8 >2014-08-19 11:06:53,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7651d1d8 >2014-08-19 11:06:53,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49a9e9c3 >2014-08-19 11:06:53,163 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [1307f0db] Discovered no tasks on Storage Pool DC1 >2014-08-19 11:06:54,497 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [32e7cf13] Running command: LogoutUserCommand internal: false. >2014-08-19 11:06:54,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [32e7cf13] Correlation ID: 32e7cf13, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 11:07:01,915 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-19 11:07:01,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 11:07:42,805 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-08-19 11:07:42,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 11:09:44,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:09:44,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:09:44,482 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:09:44,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45fce454 >2014-08-19 11:09:45,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 45fce454 >2014-08-19 11:09:48,385 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,386 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,388 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-12) [3f945f1f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:09:48,767 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,768 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:50,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:09:50,307 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:09:50,393 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:09:50,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 71f9fc3f >2014-08-19 11:09:50,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 71f9fc3f >2014-08-19 11:09:50,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: 3f945f1f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 11:37:01,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:01,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:37:01,756 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:01,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: b3c2b77 >2014-08-19 11:37:42,062 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-19 11:37:42,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: b3c2b77 >2014-08-19 11:37:42,066 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-19 11:37:42,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:42,211 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds05.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:37:42,212 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:42,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1ef9da25 >2014-08-19 11:37:42,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1ef9da25 >2014-08-19 11:37:52,741 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,754 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,763 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [7dcdbadc] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:37:52,793 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,794 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:53,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:53,079 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:37:53,111 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:53,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3f1fad32 >2014-08-19 11:38:33,551 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-19 11:38:33,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 3f1fad32 >2014-08-19 11:38:33,556 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-19 11:38:33,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:38:33,926 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds05.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:38:33,927 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:38:33,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3753356b >2014-08-19 11:38:34,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3753356b >2014-08-19 11:38:34,142 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 7dcdbadc, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 12:06:39,711 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-19 12:06:40,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-08-19 12:12:16,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-30) Command ListVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:16,322 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:16,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-19 12:12:16,348 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-19 12:12:16,457 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:16,459 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:16,460 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-19 12:12:16,476 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-19 12:12:21,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:21,091 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:12:21,096 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [45b4b475] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 12:12:21,113 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [45b4b475] Correlation ID: 45b4b475, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-19 12:12:21,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,371 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:21,372 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:21,406 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,505 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,507 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:21,508 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:21,530 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,421 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:26,502 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Server failed to respond, vds_id = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds_name = monique-vds05.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,512 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com is non responsive. >2014-08-19 12:12:26,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,546 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:26,550 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-35) ResourceManager::vdsNotResponding entered for Host 4290a94f-9e6e-4eed-a517-3401f613ebdf, monique-vds05.tlv.redhat.com >2014-08-19 12:12:26,604 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:26,715 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Server failed to respond, vds_id = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,737 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 12:12:26,742 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] Opening SSH Soft Fencing session on host monique-vds05.tlv.redhat.com >2014-08-19 12:12:26,756 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-08-19 12:12:26,786 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-3) ResourceManager::vdsNotResponding entered for Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, monique-vds01.tlv.redhat.com >2014-08-19 12:12:26,820 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:26,889 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-19 12:12:26,903 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-08-19 12:12:31,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:31,706 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:31,713 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:31,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:31,888 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:31,903 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:36,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:36,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:36,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:36,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:36,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:36,944 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:41,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:41,814 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:41,821 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:41,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:41,988 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:42,010 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:46,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:46,869 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:46,902 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:47,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:47,053 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:47,059 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:51,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:51,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:51,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:52,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:52,107 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:52,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:57,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:57,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:57,057 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:57,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:57,161 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:57,167 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:02,096 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:02,098 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:02,104 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:02,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:02,207 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:02,220 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:07,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:07,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:07,166 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:07,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:07,254 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:07,302 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:12,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:12,204 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:12,213 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:12,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:12,349 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:12,359 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:17,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:17,256 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:17,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:17,397 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:17,399 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:17,429 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:22,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:22,313 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:22,321 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:22,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:22,462 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:22,472 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:27,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:27,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:27,369 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:27,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:27,517 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:27,523 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:29,989 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] SSH Soft Fencing command failed on host monique-vds05.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view6.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-19 12:13:30,003 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] SSH Soft Fencing command failed on host monique-vds01.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view6.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-19 12:13:30,152 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [47df259c] Lock Acquired to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,154 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [47df259c] Fence failed on vds monique-vds01.tlv.redhat.com which is spm of pool ab2440ae-4286-4465-beed-c2ccc8256f8b - moving pool to non operational >2014-08-19 12:13:30,173 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] The message key SetStoragePoolStatus is missing from bundles/ExecutionMessages >2014-08-19 12:13:30,192 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Lock Acquired to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,194 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Failed to run Fence script on vds:monique-vds05.tlv.redhat.com. >2014-08-19 12:13:30,210 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-08-19 12:13:30,212 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-08-19 12:13:30,213 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Lock freed to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,215 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:13:30,238 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Correlation ID: b75ee42, Job ID: 4acbec7c-71aa-4e70-acbf-137ef75744ea, Call Stack: null, Custom Event ID: -1, Message: Fencing failed on Storage Pool Manager monique-vds01.tlv.redhat.com for Data Center DC1. Setting status to Non-Operational. >2014-08-19 12:13:30,290 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,291 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Failed to run Fence script on vds:monique-vds01.tlv.redhat.com. >2014-08-19 12:13:30,306 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-08-19 12:13:30,308 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-08-19 12:13:30,309 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Trying to release exclusive lock which does not exist, lock key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affbVDS_FENCE >2014-08-19 12:13:30,310 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:32,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:32,401 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:32,410 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:32,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:32,555 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:32,561 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:37,443 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:37,445 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:37,452 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:37,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:37,598 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:37,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:42,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:42,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:42,496 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:42,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:42,650 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:42,656 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:47,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:47,534 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:47,541 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:47,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:47,690 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:47,695 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:52,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:52,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:52,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:52,728 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:52,730 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:52,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:57,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:57,621 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:57,642 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:57,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:57,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:57,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:02,676 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:02,678 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:02,684 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:02,811 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:02,813 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:02,819 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:07,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:07,718 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:07,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:07,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:07,858 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:07,864 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:12,759 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:12,760 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:12,766 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:12,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:12,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:12,928 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:17,796 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:17,798 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:17,805 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:17,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:17,965 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:17,972 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:22,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:22,862 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:22,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:23,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:23,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:23,056 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:27,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:27,902 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:27,908 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:28,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:28,090 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:28,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:32,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:32,943 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:32,949 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:33,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:33,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:33,140 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:37,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:37,982 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:37,988 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:38,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:38,174 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:38,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:43,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:43,027 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:43,034 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:43,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:43,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:43,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:48,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:48,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:48,083 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:48,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:48,310 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:48,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:53,122 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:53,124 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:53,139 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:53,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:53,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:53,370 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:58,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:58,173 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:58,183 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:58,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:58,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:58,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:03,215 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:03,217 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:03,224 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:03,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:03,458 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:03,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:08,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:08,260 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:08,266 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:08,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:08,498 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:08,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:13,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:13,304 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:13,311 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:13,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:13,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:13,546 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:18,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:18,344 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:18,350 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:18,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:18,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:18,589 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:21,127 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] IrsBroker::Failed::GetStoragePoolInfoVDS due to: TimeoutException: >2014-08-19 12:15:23,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [45b4b475] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-19 12:15:23,273 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [3f2499db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:15:23,286 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3f2499db] Correlation ID: 3f2499db, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-19 12:15:23,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:23,318 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:23,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:23,323 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:23,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,394 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:23,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:23,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:23,633 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:28,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:28,468 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:28,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:28,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:28,681 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:28,687 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:33,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:33,433 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:33,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:33,443 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:33,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:33,558 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:33,564 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:33,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:33,725 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:33,733 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:38,598 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:38,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:38,611 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:38,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:38,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:38,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:43,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:43,534 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:43,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:43,538 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:43,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:43,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:43,662 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:43,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:43,809 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:43,824 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:48,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:48,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:48,708 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:48,855 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:48,857 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:48,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:53,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:53,629 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:53,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:53,633 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:53,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:53,751 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:53,758 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:53,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:53,907 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:53,913 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:58,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:58,794 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:58,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:58,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:58,956 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:58,962 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:03,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:03,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:03,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:03,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:03,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:03,837 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:03,844 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:03,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:03,993 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:04,000 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:08,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:08,880 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:08,886 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:09,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:09,033 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:09,039 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:13,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:13,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:13,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:13,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:13,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:13,919 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:13,927 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:14,072 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [434e9f98] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:14,073 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [434e9f98] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:14,080 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [434e9f98] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:19,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:19,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:19,035 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:19,112 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:19,114 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:19,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:23,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:23,926 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:23,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:23,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:24,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:24,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:24,079 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:24,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:24,158 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:24,193 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:29,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:29,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:29,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:29,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:29,225 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:29,231 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:34,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:34,031 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:34,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:34,035 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:34,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:34,154 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:34,160 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:34,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:34,262 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:34,321 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:39,191 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:39,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:39,201 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:39,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:39,355 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:39,363 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:44,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:44,132 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:44,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:44,136 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:44,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:44,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:44,246 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:44,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:44,402 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:44,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:49,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:49,289 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:49,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:49,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:49,441 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:49,448 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:54,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:54,240 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:54,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:54,244 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:54,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:54,453 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:54,459 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:54,560 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:54,562 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:54,568 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:59,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:59,532 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:59,539 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:59,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:59,607 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:59,656 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:04,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:04,342 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:04,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:04,346 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:04,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:04,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:04,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:04,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:04,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:04,707 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:09,640 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:09,642 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:09,649 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:09,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:09,741 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:09,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:14,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:14,439 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:14,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:14,444 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:14,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:14,691 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:14,698 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:14,779 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:14,780 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:14,792 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:19,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:19,732 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:19,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:19,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:19,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:19,835 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:24,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:24,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:24,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:24,585 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:24,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:24,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:24,790 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:24,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:24,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:24,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:29,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:29,834 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:29,842 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:29,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:29,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:29,967 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:34,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:34,676 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:34,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:34,680 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:34,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:34,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:34,885 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:35,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:35,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:35,014 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:39,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:39,930 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:39,936 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:40,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:40,053 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:40,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:44,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:44,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:44,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:44,791 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:44,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:44,977 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:44,983 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:45,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:45,095 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:45,101 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:50,013 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:50,015 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:50,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:50,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:50,143 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:50,156 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:54,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:54,884 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:54,888 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:55,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:55,069 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:55,075 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:55,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:55,192 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:55,209 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:00,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:00,107 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:00,113 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:00,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:00,265 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:00,270 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:05,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:18:05,016 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:18:05,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:18:05,019 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] spm vds is non responsive, stopping spm selection. >2014-08-19 12:18:05,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:05,146 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:05,153 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:05,305 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:05,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:05,330 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:10,191 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:10,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:10,200 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:10,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:10,381 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:10,388 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:15,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:18:15,113 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:18:15,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:18:15,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) spm vds is non responsive, stopping spm selection. >2014-08-19 12:18:15,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:15,240 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:15,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:15,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:15,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:15,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:20,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:20,298 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:20,358 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] >2014-08-21 10:48:00,341 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-21 10:48:00,354 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-21 10:48:00,354 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-21 10:48:00,355 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-21 10:48:00,356 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-21 10:48:00,357 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-21 10:48:00,358 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-21 10:48:00,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-21 10:48:00,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-21 10:48:00,360 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-21 10:48:00,361 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-21 10:48:00,361 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-21 10:48:00,362 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-21 10:48:00,363 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-21 10:48:00,376 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-21 10:48:00,377 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-21 10:48:00,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-21 10:48:00,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-21 10:48:00,379 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-21 10:48:00,380 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-21 10:48:00,380 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-21 10:48:00,381 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-21 10:48:00,382 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine". >2014-08-21 10:48:00,382 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-21 10:48:00,383 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-21 10:48:00,384 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-21 10:48:00,384 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-21 10:48:00,385 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-21 10:48:00,386 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-21 10:48:00,387 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-21 10:48:00,387 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-21 10:48:00,400 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-21 10:48:00,401 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-21 10:48:00,401 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-21 10:48:00,402 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-21 10:48:00,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-21 10:48:00,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-21 10:48:00,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-21 10:48:00,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-21 10:48:00,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-21 10:48:00,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-21 10:48:00,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-21 10:48:00,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-21 10:48:00,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-21 10:48:00,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-21 10:48:00,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-21 10:48:00,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-21 10:48:00,428 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-21 10:48:00,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-21 10:48:00,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-21 10:48:00,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-21 10:48:00,431 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-21 10:48:00,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-21 10:48:00,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-21 10:48:00,433 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-21 10:48:00,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-21 10:48:00,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-21 10:48:00,435 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-21 10:48:00,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-21 10:48:00,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-21 10:48:00,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-21 10:48:00,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt". >2014-08-21 10:48:00,438 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-21 10:48:00,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-21 10:48:00,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-21 10:48:00,458 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-21 10:48:00,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-21 10:48:00,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-21 10:48:02,760 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-21 10:48:04,336 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-21 10:48:04,348 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-21 10:48:04,801 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-21 10:48:04,801 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-21 10:48:04,806 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-21 10:48:04,807 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-21 10:48:04,808 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-21 10:48:04,884 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-21 10:48:04,885 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-21 10:48:04,898 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-21 10:48:04,900 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-21 10:48:04,903 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-21 10:48:04,932 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-21 10:48:04,980 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-21 10:48:04,982 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-21 10:48:05,005 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-21 10:48:05,235 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-21 10:48:05,320 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-21 10:48:05,365 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-21 10:48:05,367 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-21 10:48:05,368 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-21 10:48:05,369 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-21 10:48:05,370 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-21 10:48:05,505 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-21 10:48:05,506 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-21 10:48:05,507 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-21 10:48:05,509 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-21 10:48:05,512 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-21 10:48:05,514 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-21 10:48:05,765 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-21 10:48:05,766 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-21 10:48:05,767 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-21 10:48:05,769 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-21 10:48:06,107 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-21 10:48:06,119 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-21 10:48:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-21 10:48:06,209 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-21 10:48:06,455 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:06,524 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-21 10:48:06,826 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-21 10:48:07,063 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-21 10:48:07,064 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:07,082 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-21 10:48:07,111 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-21 10:48:07,111 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:07,150 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-21 10:48:07,181 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-21 10:48:07,241 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-21 10:48:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-21 10:48:07,249 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-21 10:48:07,254 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-21 10:48:07,255 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-21 10:48:07,256 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-21 10:48:07,268 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-21 10:48:07,269 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-21 10:48:07,293 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-21 10:48:07,275 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-21 10:48:07,343 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-21 10:48:07,420 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-21 10:48:07,420 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-21 10:48:07,421 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-21 10:48:07,423 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-21 10:48:11,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:11,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:11,135 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:11,137 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:11,257 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-21 10:48:11,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: UNASSIGNED not exist in string table >2014-08-21 10:48:11,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-21 10:48:11,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-21 10:48:11,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-21 10:48:11,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-21 10:48:11,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-21 10:48:11,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-21 10:48:11,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-21 10:48:11,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-21 10:48:11,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-21 10:48:11,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STOPPED not exist in string table >2014-08-21 10:48:11,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STARTED not exist in string table >2014-08-21 10:48:11,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_ERROR not exist in string table >2014-08-21 10:48:11,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-21 10:48:11,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-21 10:48:11,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-21 10:48:11,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-21 10:48:11,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:11,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:11,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:11,733 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:12,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 2ff5f506 >2014-08-21 10:48:12,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 2ff5f506 >2014-08-21 10:48:12,793 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-21 10:48:13,170 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 10:48:14,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:14,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:14,700 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:14,722 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:14,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:14,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:14,769 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:14,834 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:17,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:17,749 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:17,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:17,966 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:18,035 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Server failed to respond, vds_id = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds_name = monique-vds05.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.ConnectException: Connection refused >2014-08-21 10:48:18,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:18,150 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Server failed to respond, vds_id = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.ConnectException: Connection refused >2014-08-21 10:48:18,163 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:18,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:18,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 26e95ea3 >2014-08-21 10:48:18,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) SpmStopVDSCommand:: vds monique-vds01.tlv.redhat.com is in Connecting status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 10:48:18,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, SpmStopVDSCommand, log id: 26e95ea3 >2014-08-21 10:48:18,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) spm stop on spm failed, stopping spm selection! >2014-08-21 10:48:18,392 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com is non responsive. >2014-08-21 10:48:18,415 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-08-21 10:48:18,496 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-22) ResourceManager::vdsNotResponding entered for Host 4290a94f-9e6e-4eed-a517-3401f613ebdf, monique-vds05.tlv.redhat.com >2014-08-21 10:48:18,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:18,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:18,544 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-23) ResourceManager::vdsNotResponding entered for Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, monique-vds01.tlv.redhat.com >2014-08-21 10:48:18,773 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:18,778 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Opening SSH Soft Fencing session on host monique-vds05.tlv.redhat.com >2014-08-21 10:48:18,966 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:48:18,983 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-08-21 10:48:20,231 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-08-21 10:48:20,241 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Executing SSH Soft Fencing command on host monique-vds05.tlv.redhat.com >2014-08-21 10:48:21,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:21,756 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:21,774 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:21,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:21,797 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:21,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:26,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:26,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:26,950 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:26,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:26,973 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:27,025 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:28,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:28,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:28,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:28,450 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) spm vds is non responsive, stopping spm selection. >2014-08-21 10:48:30,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 27f4f4ce >2014-08-21 10:48:30,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GetHardwareInfoVDSCommand, log id: 27f4f4ce >2014-08-21 10:48:31,079 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:31,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 42d0cb88 >2014-08-21 10:48:31,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] FINISH, GetHardwareInfoVDSCommand, log id: 42d0cb88 >2014-08-21 10:48:31,299 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-21 10:48:31,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 467de36d >2014-08-21 10:48:31,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] FINISH, SetMOMPolicyParametersVDSCommand, log id: 467de36d >2014-08-21 10:48:31,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [20599bed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-08-21 10:48:31,542 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [20599bed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-21 10:48:31,583 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20599bed] Initializing Host: monique-vds05.tlv.redhat.com >2014-08-21 10:48:31,591 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [6f5fe43e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:32,096 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:32,097 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:32,107 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:35,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7acd9284 >2014-08-21 10:48:35,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = monique-vds01.tlv.redhat.com >2014-08-21 10:48:35,237 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-21 10:48:35,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is initializing. Message: Recovering from crash or Initializing >2014-08-21 10:48:35,668 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2014-08-21 10:48:38,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:38,572 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:38,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:38,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-90) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 6ef6fe22 >2014-08-21 10:48:38,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-08-21 10:48:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) HostName = monique-vds01.tlv.redhat.com >2014-08-21 10:48:38,668 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-08-21 10:48:38,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, SpmStopVDSCommand, log id: 6ef6fe22 >2014-08-21 10:48:38,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) spm stop succeeded, continuing with spm selection >2014-08-21 10:48:38,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId 1, LVER 5 >2014-08-21 10:48:38,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=1, prevLVER=5, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 3b72654e >2014-08-21 10:48:38,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling started: taskId = 0adefff1-d9e8-467a-8aab-02828a379473 >2014-08-21 10:48:39,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 4a0a0394 >2014-08-21 10:48:39,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 4a0a0394 >2014-08-21 10:48:39,712 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:48:39,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling ended: taskId = 0adefff1-d9e8-467a-8aab-02828a379473 task status = finished >2014-08-21 10:48:39,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: caf5747 >2014-08-21 10:48:39,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] FINISH, GetHardwareInfoVDSCommand, log id: caf5747 >2014-08-21 10:48:39,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling ended, spm status: SPM >2014-08-21 10:48:39,902 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [607b2f00] Cancelling the recovery from crash timer for VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb because vds started initializing >2014-08-21 10:48:39,905 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [395fc907] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 10:48:39,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-90) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=0adefff1-d9e8-467a-8aab-02828a379473), log id: 422b00b1 >2014-08-21 10:48:39,958 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [23375153] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 10:48:39,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, HSMClearTaskVDSCommand, log id: 422b00b1 >2014-08-21 10:48:39,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5907a8b6, log id: 3b72654e >2014-08-21 10:48:39,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 10:48:40,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63a68eed >2014-08-21 10:48:40,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 10:48:40,125 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 10:48:40,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 7296b586 >2014-08-21 10:48:40,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 10:48:40,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 46a1d6fe >2014-08-21 10:48:40,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46a1d6fe >2014-08-21 10:48:40,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7296b586 >2014-08-21 10:48:40,351 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool DC1 >2014-08-21 10:48:40,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 63a68eed >2014-08-21 10:48:40,920 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [23375153] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 10:48:40,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 3d38bcff >2014-08-21 10:48:41,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, ConnectStoragePoolVDSCommand, log id: 3d38bcff >2014-08-21 10:48:41,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 396ae4bf >2014-08-21 10:48:41,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] FINISH, SetMOMPolicyParametersVDSCommand, log id: 396ae4bf >2014-08-21 10:48:41,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [23375153] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-08-21 10:48:41,758 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [23375153] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 10:48:41,782 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [23375153] Initializing Host: monique-vds01.tlv.redhat.com >2014-08-21 10:48:41,787 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [43495829] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:49:00,972 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginAdminUserCommand internal: false. >2014-08-21 10:49:01,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 10:49:10,799 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-21 10:49:10,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 10:49:31,596 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp--127.0.0.1-8702-3) [50a32d2e] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-21 10:49:35,104 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp--127.0.0.1-8702-5) [541b55be] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-21 10:50:02,674 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:02,675 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:02,677 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [1d2e10c0] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 10:50:03,024 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:03,025 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:03,271 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 10:50:03,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 1d2e10c0, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:28,027 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-21 11:01:28,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 11:01:28,080 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [1d7ce352] Running command: LogoutBySessionCommand internal: false. >2014-08-21 11:01:28,095 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [e54cf86] Running command: LogoutUserCommand internal: false. >2014-08-21 11:01:28,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [e54cf86] Correlation ID: e54cf86, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-21 11:01:38,278 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-21 11:01:38,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 11:01:41,075 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:41,088 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:41,089 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_a5eb1b3a-12f4-409c] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT_CREDENTIALS >2014-08-21 11:01:41,125 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but Agent credentials are missing.] >2014-08-21 11:01:43,313 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,314 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,315 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_98696063-2f44-48ae] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:43,386 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,387 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,520 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:43,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_98696063-2f44-48ae, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:45,375 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,376 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,388 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_7f4f56e2-55b9-4b13] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:45,409 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,410 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,464 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:45,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_7f4f56e2-55b9-4b13, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:46,177 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,178 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,205 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_2a1978a8-6b0f-4127] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT_CREDENTIALS >2014-08-21 11:01:46,207 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but Agent credentials are missing.] >2014-08-21 11:01:46,956 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,957 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,958 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_ec8262e7-74ad-4005] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:46,989 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,990 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,037 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:47,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_ec8262e7-74ad-4005, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:47,328 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,329 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,330 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_3b3c7bc9-77e9-40bd] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:47,367 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,415 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:47,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_3b3c7bc9-77e9-40bd, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:48,077 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,077 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,078 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_e39f10a6-efbd-484d] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT >2014-08-21 11:01:48,080 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but no Agent type selected.] >2014-08-21 11:01:48,753 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,753 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,754 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_154ae1e2-f951-4d60] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:48,801 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,802 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,838 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:48,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_154ae1e2-f951-4d60, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:49,134 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,135 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,136 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_bf231ac4-804d-4af0] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:49,172 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,173 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,219 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:49,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_bf231ac4-804d-4af0, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:49,844 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,845 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,846 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_39680d7a-2801-4adc] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_AGENT_NOT_SUPPORTED >2014-08-21 11:01:49,847 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Selected Power Management Agent is not supported.] >2014-08-21 11:01:50,288 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,289 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,290 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_e390248d-1a87-4d40] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:50,296 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,297 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,324 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:50,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_e390248d-1a87-4d40, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:50,514 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,515 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,516 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_2c61516a-78ec-4eb8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:50,524 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,525 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,553 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:50,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_2c61516a-78ec-4eb8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:51,101 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,102 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,103 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_1bc318ee-0f24-4841] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:51,172 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,173 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:51,574 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:01:51,613 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:01:51,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 29004677 >2014-08-21 11:01:52,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 29004677 >2014-08-21 11:01:52,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_1bc318ee-0f24-4841, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:02:00,736 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:00,977 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:02:01,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Correlation ID: vms_create_5f3cb986-173e-480b, Job ID: 2586dd42-6099-4f5b-b107-28c0b581d609, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was created by admin. >2014-08-21 11:02:01,685 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Lock freed to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:08,550 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-4) [nics_create_89fea944-58d1-4cc6] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:02:08,653 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-4) [d2c325f] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:02:08,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [d2c325f] Correlation ID: d2c325f, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_ha. (User: admin) >2014-08-21 11:02:08,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [d2c325f] Correlation ID: nics_create_89fea944-58d1-4cc6, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_ha. (User: admin) >2014-08-21 11:02:12,419 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM_DISK_BOOT >, sharedLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >] >2014-08-21 11:02:12,488 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Running command: AddDiskCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:12,489 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_ha_Disk1 >2014-08-21 11:02:12,876 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:13,029 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM_DISK_BOOT >, sharedLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >] >2014-08-21 11:02:13,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = 46aa71cf-39b1-4d85-a28b-41cddf3c9315, newImageDescription = ), log id: e1a9f15 >2014-08-21 11:02:13,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:02:13,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] FINISH, CreateImageVDSCommand, return: 46aa71cf-39b1-4d85-a28b-41cddf3c9315, log id: e1a9f15 >2014-08-21 11:02:13,476 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [2e26c3b4] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 3c7421d2-0b83-458a-b920-5417c9499dfd >2014-08-21 11:02:13,477 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [2e26c3b4] CommandMultiAsyncTasks::AttachTask: Attaching task 07c43944-f88f-4e80-abfd-ce959107ca88 to command 3c7421d2-0b83-458a-b920-5417c9499dfd. >2014-08-21 11:02:13,526 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [2e26c3b4] Adding task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:02:13,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [2e26c3b4] Correlation ID: disks_create_4bca8449-2ee1-4ca6, Job ID: 28d89b8a-1e66-4b03-8dde-b5f31e90cd9b, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_ha_Disk1 was initiated on VM vm_ha by admin. >2014-08-21 11:02:13,610 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [2e26c3b4] BaseAsyncTask::startPollingTask: Starting to poll task 07c43944-f88f-4e80-abfd-ce959107ca88. >2014-08-21 11:02:16,265 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-95) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-08-21 11:02:16,287 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-95) SPMAsyncTask::PollTask: Polling task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:02:16,296 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-95) BaseAsyncTask::OnTaskEndSuccess: Task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:02:16,297 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-95) CommandAsyncTask::EndActionIfNecessary: All tasks of command 3c7421d2-0b83-458a-b920-5417c9499dfd has ended -> executing endAction >2014-08-21 11:02:16,298 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-95) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 3c7421d2-0b83-458a-b920-5417c9499dfd): calling endAction . >2014-08-21 11:02:16,300 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:02:16,318 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-45) [disks_create_4bca8449-2ee1-4ca6] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:02:16,338 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:02:16,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, imageId = 46aa71cf-39b1-4d85-a28b-41cddf3c9315), log id: 11616204 >2014-08-21 11:02:16,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@f03cb394, log id: 11616204 >2014-08-21 11:02:16,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) Correlation ID: disks_create_4bca8449-2ee1-4ca6, Call Stack: null, Custom Event ID: -1, Message: The disk vm_ha_Disk1 was successfully added to VM vm_ha. >2014-08-21 11:02:16,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:02:16,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:02:16,410 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-45) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 07c43944-f88f-4e80-abfd-ce959107ca88 >2014-08-21 11:02:16,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 07c43944-f88f-4e80-abfd-ce959107ca88), log id: 7d75be37 >2014-08-21 11:02:16,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=07c43944-f88f-4e80-abfd-ce959107ca88), log id: 3d1b4d3a >2014-08-21 11:02:16,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, HSMClearTaskVDSCommand, log id: 3d1b4d3a >2014-08-21 11:02:16,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, SPMClearTaskVDSCommand, log id: 7d75be37 >2014-08-21 11:02:16,489 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-45) BaseAsyncTask::RemoveTaskFromDB: Removed task 07c43944-f88f-4e80-abfd-ce959107ca88 from DataBase >2014-08-21 11:02:16,490 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 3c7421d2-0b83-458a-b920-5417c9499dfd >2014-08-21 11:02:25,488 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:02:25,773 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, IsVmDuringInitiatingVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75), log id: 6e2e2e52 >2014-08-21 11:02:25,774 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6e2e2e52 >2014-08-21 11:02:26,003 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Running command: RunVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:02:26,352 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@482bb400), log id: 6e9a15b2 >2014-08-21 11:02:26,535 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6e9a15b2 >2014-08-21 11:02:26,599 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 1cdba630 >2014-08-21 11:02:26,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 2fbf0907 >2014-08-21 11:02:27,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiNmFkMWZkMzItYTE0NC00MmYw >LThmYWYtNzk4YmQ2NDM4YjVhIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=2fa59eb2-1abe-4dba-a2d6-3a1534d648d3}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=063dada5-e1f8-4d26-a1a6-3f36e9dbaeaa}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8a8b77f4-9743-4fde-a10a-e964561baeaf}, {shared=false, volumeID=46aa71cf-39b1-4d85-a28b-41cddf3c9315, index=0, iface=virtio, imageID=0045c380-09dd-4340-ac3a-b22edee60ca8, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=0045c380-09dd-4340-ac3a-b22edee60ca8}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=45954a54-2239-49be-abec-0db12b68a8ae}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=54a9ed8e-e144-40f4-a409-21ad16885778}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=8a1d6f2c-cfae-4c6a-b1ab-2f288c07e9b8}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:02:27,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2fbf0907 >2014-08-21 11:02:27,438 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1cdba630 >2014-08-21 11:02:27,438 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:02:27,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Correlation ID: vms_syncAction_9dc5ee87-4dc6-474d, Job ID: c13c1f72-b942-4383-9ab7-654de767ef9c, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:02:29,634 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:02:29,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-42) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[64ff4f66-e697-4a93-b7ff-dba82eb02b75]), log id: 260020d >2014-08-21 11:02:29,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=8055, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@6d26a4b2, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 260020d >2014-08-21 11:02:29,683 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Received a spice Device without an address when processing VM 64ff4f66-e697-4a93-b7ff-dba82eb02b75 devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:02:40,116 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:40,342 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:02:40,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Correlation ID: vms_create_84d3bbe4-566e-4a0e, Job ID: 0296d326-ab1b-401a-8cc0-fd241f41fbac, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was created by admin. >2014-08-21 11:02:40,551 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Lock freed to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:41,842 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-6) [nics_create_07c7acc3-03d2-4341] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:02:41,871 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-6) [5f257142] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:02:41,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f257142] Correlation ID: 5f257142, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_2. (User: admin) >2014-08-21 11:02:41,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f257142] Correlation ID: nics_create_07c7acc3-03d2-4341, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_2. (User: admin) >2014-08-21 11:02:42,813 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM_DISK_BOOT >, sharedLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >] >2014-08-21 11:02:42,843 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Running command: AddDiskCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:42,845 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_2_Disk1 >2014-08-21 11:02:42,878 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [492dc420] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:42,887 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [492dc420] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM_DISK_BOOT >, sharedLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >] >2014-08-21 11:02:42,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = e44ee009-3a78-40e2-b024-e236b205a702, newImageDescription = ), log id: 7bcebfdc >2014-08-21 11:02:42,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:02:42,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] FINISH, CreateImageVDSCommand, return: e44ee009-3a78-40e2-b024-e236b205a702, log id: 7bcebfdc >2014-08-21 11:02:42,975 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [492dc420] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 58878f92-8d69-4c7b-8275-678acfd01cfc >2014-08-21 11:02:42,976 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [492dc420] CommandMultiAsyncTasks::AttachTask: Attaching task 6dba1c95-354d-4318-bb82-d1658b766252 to command 58878f92-8d69-4c7b-8275-678acfd01cfc. >2014-08-21 11:02:43,017 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [492dc420] Adding task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:02:43,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [492dc420] Correlation ID: disks_create_a6425d58-a616-44b8, Job ID: ac69d713-b229-4047-b594-b2876b141799, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_2_Disk1 was initiated on VM vm_2 by admin. >2014-08-21 11:02:43,084 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [492dc420] BaseAsyncTask::startPollingTask: Starting to poll task 6dba1c95-354d-4318-bb82-d1658b766252. >2014-08-21 11:02:46,305 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-93) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now >2014-08-21 11:02:46,310 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) SPMAsyncTask::PollTask: Polling task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:02:46,328 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) BaseAsyncTask::OnTaskEndSuccess: Task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:02:46,329 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::EndActionIfNecessary: All tasks of command 58878f92-8d69-4c7b-8275-678acfd01cfc has ended -> executing endAction >2014-08-21 11:02:46,330 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 58878f92-8d69-4c7b-8275-678acfd01cfc): calling endAction . >2014-08-21 11:02:46,331 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:02:46,347 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-33) [disks_create_a6425d58-a616-44b8] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:02:46,372 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:02:46,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, imageId = e44ee009-3a78-40e2-b024-e236b205a702), log id: 52c24bbc >2014-08-21 11:02:46,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8c462649, log id: 52c24bbc >2014-08-21 11:02:46,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) Correlation ID: disks_create_a6425d58-a616-44b8, Call Stack: null, Custom Event ID: -1, Message: The disk vm_2_Disk1 was successfully added to VM vm_2. >2014-08-21 11:02:46,420 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:02:46,421 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:02:46,424 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 6dba1c95-354d-4318-bb82-d1658b766252 >2014-08-21 11:02:46,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 6dba1c95-354d-4318-bb82-d1658b766252), log id: 26dcf806 >2014-08-21 11:02:46,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=6dba1c95-354d-4318-bb82-d1658b766252), log id: 605783fa >2014-08-21 11:02:46,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMClearTaskVDSCommand, log id: 605783fa >2014-08-21 11:02:46,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMClearTaskVDSCommand, log id: 26dcf806 >2014-08-21 11:02:46,502 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) BaseAsyncTask::RemoveTaskFromDB: Removed task 6dba1c95-354d-4318-bb82-d1658b766252 from DataBase >2014-08-21 11:02:46,503 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 58878f92-8d69-4c7b-8275-678acfd01cfc >2014-08-21 11:02:53,604 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:02:53,645 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, IsVmDuringInitiatingVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330), log id: 6fd9c9f5 >2014-08-21 11:02:53,656 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6fd9c9f5 >2014-08-21 11:02:53,779 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Running command: RunVmCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:02:53,992 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@ddc8f42b), log id: 696ffcaf >2014-08-21 11:02:54,026 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, UpdateVmDynamicDataVDSCommand, log id: 696ffcaf >2014-08-21 11:02:54,095 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, vm=VM [vm_2]), log id: 2a5556b0 >2014-08-21 11:02:54,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, vm=VM [vm_2]), log id: 2cab4844 >2014-08-21 11:02:54,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiZmE5YmU3OTMtNzFhMC00MmI1 >LThiZGItMzBhM2IwMDhhZjE1IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=51d5d3f5-e402-4242-81a6-f24bb8c563c8}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=1c30cb18-db5c-473f-9f6c-64aaaf2f4983}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=0e4b0a84-54d5-4045-92ad-a858d81fad19}, {shared=false, volumeID=e44ee009-3a78-40e2-b024-e236b205a702, index=0, iface=virtio, imageID=3b7c6a02-e649-4b99-91a0-62e15e88e85e, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=3b7c6a02-e649-4b99-91a0-62e15e88e85e}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=316c1233-4d21-48be-885d-a0bafb231cc9}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=4bdd9ddc-8f75-4130-b070-bca70ecee19b}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=5ef82c7e-442b-40d0-89bb-a0c66e778d62}],vmName=vm_2,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:02:54,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2cab4844 >2014-08-21 11:02:54,296 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2a5556b0 >2014-08-21 11:02:54,297 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:02:54,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Correlation ID: vms_syncAction_3f06bbf1-f98f-4299, Job ID: 3fa2d29f-4b21-4dbb-8d5f-c340f93c54fd, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:02:57,942 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) VM vm_2 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:02:57,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-29) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[8e7744c0-9b41-4db0-a62c-d85d7ddaa330]), log id: 3aabcd03 >2014-08-21 11:02:57,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=8237, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@65209e0c, vmName=vm_2, fileTransferEnable=true, cpuType=Conroe}], log id: 3aabcd03 >2014-08-21 11:02:57,973 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Received a spice Device without an address when processing VM 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5903, type=graphics} >2014-08-21 11:03:07,567 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_dc5f5594-bf04-4869] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:03:07,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:07,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:07,801 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:07,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:07,920 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:07,963 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:07,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3fdbcf45 >2014-08-21 11:03:08,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3fdbcf45 >2014-08-21 11:03:08,239 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:08,283 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 65ed7a8 >2014-08-21 11:03:08,284 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 11:03:08,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 20b810dc >2014-08-21 11:03:08,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 518ac31f >2014-08-21 11:03:08,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:03:08,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SpmStopVDSCommand, log id: 518ac31f >2014-08-21 11:03:08,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, ResetIrsVDSCommand, log id: 20b810dc >2014-08-21 11:03:08,399 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SetVdsStatusVDSCommand, log id: 65ed7a8 >2014-08-21 11:03:08,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:08,516 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:08,549 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:08,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7d8959ef >2014-08-21 11:03:08,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, FenceVdsVDSCommand, return: Done, log id: 7d8959ef >2014-08-21 11:03:08,680 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:03:13,682 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:03:13,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:13,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:13,796 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:13,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3af20533 >2014-08-21 11:03:13,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 3af20533 >2014-08-21 11:03:13,927 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:03:13,930 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SetVmStatusVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75, status = Down, exit status = Error), log id: 7a4d95ee >2014-08-21 11:03:13,947 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SetVmStatusVDSCommand, log id: 7a4d95ee >2014-08-21 11:03:13,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_ha was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:03:13,997 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:03:14,030 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] START, SetVmStatusVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330, status = Down, exit status = Error), log id: 3804f5ec >2014-08-21 11:03:14,038 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] FINISH, SetVmStatusVDSCommand, log id: 3804f5ec >2014-08-21 11:03:14,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [7ea335eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_2 was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:03:14,102 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:03:14,177 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 5dc89493 >2014-08-21 11:03:14,196 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 5dc89493 >2014-08-21 11:03:14,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [30a8f4dc] Correlation ID: 270799bf, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:03:14,304 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [6d728b03] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:14,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:14,592 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:14,597 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:14,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5cec8ccf >2014-08-21 11:03:14,676 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 5cec8ccf >2014-08-21 11:03:14,748 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, IsVmDuringInitiatingVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75), log id: 19d16c91 >2014-08-21 11:03:14,750 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 19d16c91 >2014-08-21 11:03:14,852 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] Running command: RunVmCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:03:14,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:14,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:14,923 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, UpdateVmDynamicDataVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@25271673), log id: 7554c86e >2014-08-21 11:03:14,972 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:14,974 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7554c86e >2014-08-21 11:03:14,987 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, CreateVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 2cf6f5d1 >2014-08-21 11:03:14,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5deadebd >2014-08-21 11:03:14,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, CreateVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 6ca863f >2014-08-21 11:03:15,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=063dada5-e1f8-4d26-a1a6-3f36e9dbaeaa}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8a8b77f4-9743-4fde-a10a-e964561baeaf}, {shared=false, index=0, volumeID=46aa71cf-39b1-4d85-a28b-41cddf3c9315, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, imageID=0045c380-09dd-4340-ac3a-b22edee60ca8, specParams={}, optional=false, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, readonly=false, deviceId=0045c380-09dd-4340-ac3a-b22edee60ca8}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=45954a54-2239-49be-abec-0db12b68a8ae}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=54a9ed8e-e144-40f4-a409-21ad16885778}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=8a1d6f2c-cfae-4c6a-b1ab-2f288c07e9b8}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:03:15,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 5deadebd >2014-08-21 11:03:15,200 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:03:15,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, CreateVDSCommand, log id: 6ca863f >2014-08-21 11:03:15,253 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2cf6f5d1 >2014-08-21 11:03:15,254 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:03:15,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1dae0525] Correlation ID: 1dae0525, Job ID: 5df08973-9395-4cd0-b6ff-444eb824ad60, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:03:17,844 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:03:17,850 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) [799af561] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 11:03:17,894 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [799af561] Correlation ID: 799af561, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:03:17,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:03:17,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:03:17,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a02669c >2014-08-21 11:03:18,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling started: taskId = 157eeb64-27cb-4528-a3f9-8bd64a5ccf31 >2014-08-21 11:03:18,381 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:03:18,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-9) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb], vmIds=[64ff4f66-e697-4a93-b7ff-dba82eb02b75]), log id: 720b036c >2014-08-21 11:03:18,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-9) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.2.45, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=16360, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@66cf007, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 720b036c >2014-08-21 11:03:18,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Received a spice Device without an address when processing VM 64ff4f66-e697-4a93-b7ff-dba82eb02b75 devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.2.45}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:03:19,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling ended: taskId = 157eeb64-27cb-4528-a3f9-8bd64a5ccf31 task status = finished >2014-08-21 11:03:19,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling ended, spm status: SPM >2014-08-21 11:03:19,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=157eeb64-27cb-4528-a3f9-8bd64a5ccf31), log id: 1b8bf2e9 >2014-08-21 11:03:19,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] FINISH, HSMClearTaskVDSCommand, log id: 1b8bf2e9 >2014-08-21 11:03:19,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29b826cf, log id: 5a02669c >2014-08-21 11:03:19,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-21 11:03:19,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [799af561] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-21 11:03:19,148 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:03:19,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 1b42a7d9 >2014-08-21 11:03:19,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:03:19,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 4356d50d >2014-08-21 11:03:19,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4356d50d >2014-08-21 11:03:19,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b42a7d9 >2014-08-21 11:03:19,235 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [799af561] Discovered no tasks on Storage Pool DC1 >2014-08-21 11:03:20,201 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:03:20,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:20,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:20,389 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:20,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 64a7bffa >2014-08-21 11:03:20,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 64a7bffa >2014-08-21 11:03:20,522 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:03:20,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 681e3c44, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:03:20,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_dc5f5594-bf04-4869, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:03:20,546 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:03:36,200 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Setting new tasks map. The map contains now 1 tasks >2014-08-21 11:04:06,201 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:04:06,202 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:04:16,334 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from PoweringUp --> Up >2014-08-21 11:04:16,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: 1dae0525, Job ID: 5df08973-9395-4cd0-b6ff-444eb824ad60, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:04:16,347 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-12) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:04:59,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:04:59,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:04:59,420 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:05:03,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6ade3f33 >2014-08-21 11:05:03,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 6ade3f33 >2014-08-21 11:05:03,623 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [3b9cb023] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:05:03,634 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [2505d1b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:05:03,643 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:05:03,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b5f814d >2014-08-21 11:05:03,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5b5f814d >2014-08-21 11:05:03,910 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:05:03,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 3e53e374 >2014-08-21 11:05:04,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 3e53e374 >2014-08-21 11:05:04,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:05:04,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [285aee97] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:05:04,318 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [285aee97] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:05:04,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5b7c22a0 >2014-08-21 11:05:04,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5b7c22a0 >2014-08-21 11:05:04,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 43193aa7 >2014-08-21 11:05:04,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, SetMOMPolicyParametersVDSCommand, log id: 43193aa7 >2014-08-21 11:05:04,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:05:04,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: 2505d1b2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:05:04,800 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:22,543 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,544 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,545 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_e9d8372c-4237-4750] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:22,551 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,552 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,608 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:06:22,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_update_e9d8372c-4237-4750, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:06:23,047 INFO [org.ovirt.engine.core.bll.StopVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] Running command: StopVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:23,053 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] START, DestroyVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, force=false, secondsToWait=0, gracefully=false, reason=), log id: 6cf12495 >2014-08-21 11:06:23,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] START, DestroyVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3700dcc4 >2014-08-21 11:06:23,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] FINISH, DestroyVDSCommand, log id: 3700dcc4 >2014-08-21 11:06:23,570 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] FINISH, DestroyVmVDSCommand, return: Down, log id: 6cf12495 >2014-08-21 11:06:23,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] Correlation ID: vms_syncAction_dedb8fb6-f209-4a68, Job ID: 3308492d-de05-409d-8ba9-fcf0f660e780, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha powered off by admin (Host: monique-vds01.tlv.redhat.com) (Reason: Not Specified). >2014-08-21 11:06:23,678 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-29) [f207e07] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:06:23,953 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:06:23,980 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Running command: RemoveVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:23,984 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] START, SetVmStatusVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75, status = ImageLocked, exit status = Normal), log id: 3db87b61 >2014-08-21 11:06:23,999 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] FINISH, SetVmStatusVDSCommand, log id: 3db87b61 >2014-08-21 11:06:24,037 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:06:24,074 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-3) [4f0d7561] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:24,097 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:06:24,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, postZeros = false, forceDelete = false), log id: 1df21d6e >2014-08-21 11:06:24,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] FINISH, DeleteImageGroupVDSCommand, log id: 1df21d6e >2014-08-21 11:06:24,483 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [2bc6f28a] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 1728d2e1-9c68-45d4-93ee-6da4e96ab872 >2014-08-21 11:06:24,484 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [2bc6f28a] CommandMultiAsyncTasks::AttachTask: Attaching task 475de8eb-b6f7-4096-813c-69678db7fa4d to command 1728d2e1-9c68-45d4-93ee-6da4e96ab872. >2014-08-21 11:06:24,583 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [2bc6f28a] Adding task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:24,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [2bc6f28a] Correlation ID: vms_delete_8de37fce-0e67-49f2, Job ID: 0cbadf15-a454-49b2-a94f-4354ef542b53, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was successfully removed. >2014-08-21 11:06:24,775 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [2bc6f28a] BaseAsyncTask::startPollingTask: Starting to poll task 475de8eb-b6f7-4096-813c-69678db7fa4d. >2014-08-21 11:06:25,149 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:06:25,179 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Running command: RemoveVmCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:06:25,180 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] START, SetVmStatusVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330, status = ImageLocked, exit status = Normal), log id: 47a40e4f >2014-08-21 11:06:25,183 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] FINISH, SetVmStatusVDSCommand, log id: 47a40e4f >2014-08-21 11:06:25,185 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:06:25,201 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-1) [61bce53] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:06:25,205 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-1) [3c3e33] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:06:25,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-1) [3c3e33] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, postZeros = false, forceDelete = false), log id: 7967e579 >2014-08-21 11:06:25,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-1) [3c3e33] FINISH, DeleteImageGroupVDSCommand, log id: 7967e579 >2014-08-21 11:06:25,391 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [3c3e33] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 >2014-08-21 11:06:25,392 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [3c3e33] CommandMultiAsyncTasks::AttachTask: Attaching task e7abe633-747a-4cdd-9ace-e41ecf623c66 to command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71. >2014-08-21 11:06:25,445 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [3c3e33] Adding task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:25,653 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3c3e33] Correlation ID: vms_delete_87782caf-2d36-4d4b, Job ID: f94b3181-424c-45dd-b0f5-9185b659d077, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was successfully removed. >2014-08-21 11:06:25,654 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [3c3e33] BaseAsyncTask::startPollingTask: Starting to poll task e7abe633-747a-4cdd-9ace-e41ecf623c66. >2014-08-21 11:06:26,041 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,042 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,043 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_48264477-b9d4-46d8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:26,108 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,109 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,347 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-08-21 11:06:26,357 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:06:26,358 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:06:26,360 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:26,394 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:06:26,395 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) BaseAsyncTask::OnTaskEndSuccess: Task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:26,396 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::EndActionIfNecessary: All tasks of command 1728d2e1-9c68-45d4-93ee-6da4e96ab872 has ended -> executing endAction >2014-08-21 11:06:26,397 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 1728d2e1-9c68-45d4-93ee-6da4e96ab872): calling endAction . >2014-08-21 11:06:26,398 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:26,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:06:26,426 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) BaseAsyncTask::OnTaskEndSuccess: Task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:26,427 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::EndActionIfNecessary: All tasks of command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 has ended -> executing endAction >2014-08-21 11:06:26,428 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 918c6e62-1e21-4c2a-8a4d-beb1a63cad71): calling endAction . >2014-08-21 11:06:26,439 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:06:26,441 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:06:26,444 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:06:26,444 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:06:26,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3fba3289 >2014-08-21 11:06:26,455 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 475de8eb-b6f7-4096-813c-69678db7fa4d >2014-08-21 11:06:26,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 475de8eb-b6f7-4096-813c-69678db7fa4d), log id: 2bf402d0 >2014-08-21 11:06:26,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=475de8eb-b6f7-4096-813c-69678db7fa4d), log id: 901d65c >2014-08-21 11:06:26,496 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:06:26,497 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:06:26,498 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:06:26,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] FINISH, HSMClearTaskVDSCommand, log id: 901d65c >2014-08-21 11:06:26,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] FINISH, SPMClearTaskVDSCommand, log id: 2bf402d0 >2014-08-21 11:06:26,564 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task e7abe633-747a-4cdd-9ace-e41ecf623c66 >2014-08-21 11:06:26,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = e7abe633-747a-4cdd-9ace-e41ecf623c66), log id: 1eb017ca >2014-08-21 11:06:26,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=e7abe633-747a-4cdd-9ace-e41ecf623c66), log id: 3ae9de83 >2014-08-21 11:06:26,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] FINISH, HSMClearTaskVDSCommand, log id: 3ae9de83 >2014-08-21 11:06:26,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] FINISH, SPMClearTaskVDSCommand, log id: 1eb017ca >2014-08-21 11:06:26,632 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] BaseAsyncTask::RemoveTaskFromDB: Removed task 475de8eb-b6f7-4096-813c-69678db7fa4d from DataBase >2014-08-21 11:06:26,633 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 1728d2e1-9c68-45d4-93ee-6da4e96ab872 >2014-08-21 11:06:26,645 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] BaseAsyncTask::RemoveTaskFromDB: Removed task e7abe633-747a-4cdd-9ace-e41ecf623c66 from DataBase >2014-08-21 11:06:26,645 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 >2014-08-21 11:06:26,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3fba3289 >2014-08-21 11:06:26,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_48264477-b9d4-46d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:06:27,722 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:27,822 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:06:27,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Correlation ID: vms_create_ff88116a-b971-40b5, Job ID: d080ee01-2a6a-4952-bb10-a963e06fd570, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was created by admin. >2014-08-21 11:06:27,972 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Lock freed to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:28,937 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-7) [nics_create_bd9ab51d-a322-41b5] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:06:28,990 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-7) [4171ca3d] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:06:29,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4171ca3d] Correlation ID: 4171ca3d, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_ha. (User: admin) >2014-08-21 11:06:29,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4171ca3d] Correlation ID: nics_create_bd9ab51d-a322-41b5, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_ha. (User: admin) >2014-08-21 11:06:29,491 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM_DISK_BOOT >, sharedLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >] >2014-08-21 11:06:29,510 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Running command: AddDiskCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:29,511 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_ha_Disk1 >2014-08-21 11:06:29,612 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-6) [2cd61457] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:29,620 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-6) [2cd61457] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM_DISK_BOOT >, sharedLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >] >2014-08-21 11:06:29,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = de7bf293-5790-426c-9f8a-c391b1396c8a, newImageDescription = ), log id: 70929ab6 >2014-08-21 11:06:29,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:06:29,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] FINISH, CreateImageVDSCommand, return: de7bf293-5790-426c-9f8a-c391b1396c8a, log id: 70929ab6 >2014-08-21 11:06:29,768 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-6) [2cd61457] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command c15aff19-5992-40de-9c84-1e24980ea3d7 >2014-08-21 11:06:29,769 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-6) [2cd61457] CommandMultiAsyncTasks::AttachTask: Attaching task ed0aef1f-d35a-45a8-88af-cb265e908409 to command c15aff19-5992-40de-9c84-1e24980ea3d7. >2014-08-21 11:06:29,784 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-6) [2cd61457] Adding task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:29,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [2cd61457] Correlation ID: disks_create_2b55890d-1f35-46db, Job ID: c9ceedf9-2878-4cf3-8bbb-3f72b8710763, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_ha_Disk1 was initiated on VM vm_ha by admin. >2014-08-21 11:06:29,800 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-6) [2cd61457] BaseAsyncTask::startPollingTask: Starting to poll task ed0aef1f-d35a-45a8-88af-cb265e908409. >2014-08-21 11:06:36,429 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-23) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-08-21 11:06:36,439 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) SPMAsyncTask::PollTask: Polling task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:36,444 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) BaseAsyncTask::OnTaskEndSuccess: Task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:36,445 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::EndActionIfNecessary: All tasks of command c15aff19-5992-40de-9c84-1e24980ea3d7 has ended -> executing endAction >2014-08-21 11:06:36,446 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: c15aff19-5992-40de-9c84-1e24980ea3d7): calling endAction . >2014-08-21 11:06:36,447 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:06:36,456 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-18) [disks_create_2b55890d-1f35-46db] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:06:36,500 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:06:36,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, imageId = de7bf293-5790-426c-9f8a-c391b1396c8a), log id: 5a005097 >2014-08-21 11:06:36,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8c694cf4, log id: 5a005097 >2014-08-21 11:06:36,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: disks_create_2b55890d-1f35-46db, Call Stack: null, Custom Event ID: -1, Message: The disk vm_ha_Disk1 was successfully added to VM vm_ha. >2014-08-21 11:06:36,668 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:06:36,669 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:06:36,675 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-18) SPMAsyncTask::ClearAsyncTask: Attempting to clear task ed0aef1f-d35a-45a8-88af-cb265e908409 >2014-08-21 11:06:36,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = ed0aef1f-d35a-45a8-88af-cb265e908409), log id: 5dfb6108 >2014-08-21 11:06:36,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=ed0aef1f-d35a-45a8-88af-cb265e908409), log id: 229f846c >2014-08-21 11:06:36,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, HSMClearTaskVDSCommand, log id: 229f846c >2014-08-21 11:06:36,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, SPMClearTaskVDSCommand, log id: 5dfb6108 >2014-08-21 11:06:36,745 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-18) BaseAsyncTask::RemoveTaskFromDB: Removed task ed0aef1f-d35a-45a8-88af-cb265e908409 from DataBase >2014-08-21 11:06:36,745 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity c15aff19-5992-40de-9c84-1e24980ea3d7 >2014-08-21 11:06:40,318 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:06:40,347 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, IsVmDuringInitiatingVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b), log id: 34bd7e7e >2014-08-21 11:06:40,347 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 34bd7e7e >2014-08-21 11:06:40,407 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Running command: RunVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:06:40,524 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@dc02e6ca), log id: ec8e3b0 >2014-08-21 11:06:40,527 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, UpdateVmDynamicDataVDSCommand, log id: ec8e3b0 >2014-08-21 11:06:40,579 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 68717012 >2014-08-21 11:06:40,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 7c00a846 >2014-08-21 11:06:40,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiYjQ2OTA2NjQtZDEyOC00ZDli >LWFhYWItNGYxZjM3MzIzNzVlIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=ac16ab88-e733-4373-bc4f-0dbec2b424b6}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=4efe0fe5-f86d-4359-a0b8-f6a1cc19feb4}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=080970ae-d7bb-494c-a4fd-832420935d2a}, {shared=false, volumeID=de7bf293-5790-426c-9f8a-c391b1396c8a, index=0, iface=virtio, imageID=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=5db0d96e-73dc-4370-a7fd-9b8f19330e3a}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=c15b2bc7-299d-4376-b060-fcf9ee1bca2c}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=4493feb7-855b-4bb5-9dd3-828324951568}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:06:40,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, CreateVmFromCloudInitVDSCommand, log id: 7c00a846 >2014-08-21 11:06:40,775 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 68717012 >2014-08-21 11:06:40,776 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:06:40,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Correlation ID: vms_syncAction_2ed08e1f-1103-44d1, Job ID: 9cff69db-1b74-461c-b66e-b954267899cf, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:06:42,840 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from WaitForLaunch --> PoweringUp >2014-08-21 11:06:42,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-44) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[04545787-d6d5-4b4d-a06f-2d42db7c933b]), log id: 6f4cacd0 >2014-08-21 11:06:42,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=4695, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@12e5c4c6, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 6f4cacd0 >2014-08-21 11:06:42,853 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Received a spice Device without an address when processing VM 04545787-d6d5-4b4d-a06f-2d42db7c933b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:06:52,125 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:52,238 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:06:52,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Correlation ID: vms_create_64d6c1c7-12b0-43f6, Job ID: a6d4d145-c47d-452f-9990-30569e8ce4b2, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was created by admin. >2014-08-21 11:06:52,314 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Lock freed to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:53,176 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-5) [nics_create_906646f7-ed97-489e] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:06:53,221 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-5) [7293acbc] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:06:53,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7293acbc] Correlation ID: 7293acbc, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_2. (User: admin) >2014-08-21 11:06:53,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7293acbc] Correlation ID: nics_create_906646f7-ed97-489e, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_2. (User: admin) >2014-08-21 11:06:53,709 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM_DISK_BOOT >, sharedLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >] >2014-08-21 11:06:53,722 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Running command: AddDiskCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:53,724 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_2_Disk1 >2014-08-21 11:06:53,766 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [59c24efb] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:53,805 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [59c24efb] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM_DISK_BOOT >, sharedLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >] >2014-08-21 11:06:53,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, newImageDescription = ), log id: 22f19931 >2014-08-21 11:06:53,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:06:53,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] FINISH, CreateImageVDSCommand, return: a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, log id: 22f19931 >2014-08-21 11:06:53,968 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [59c24efb] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 51674dc7-7774-4c78-8bb6-301f046ea07b >2014-08-21 11:06:53,980 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [59c24efb] CommandMultiAsyncTasks::AttachTask: Attaching task 73c605d7-0794-4766-ad42-b045d5bc24a8 to command 51674dc7-7774-4c78-8bb6-301f046ea07b. >2014-08-21 11:06:54,043 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [59c24efb] Adding task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:54,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [59c24efb] Correlation ID: disks_create_04957bf8-6fd6-4ba3, Job ID: 60495144-0d4b-43a9-8957-9e0352bf0332, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_2_Disk1 was initiated on VM vm_2 by admin. >2014-08-21 11:06:54,063 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) [59c24efb] BaseAsyncTask::startPollingTask: Starting to poll task 73c605d7-0794-4766-ad42-b045d5bc24a8. >2014-08-21 11:06:56,448 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-88) Polling and updating Async Tasks: 4 tasks, 1 tasks to poll now >2014-08-21 11:06:56,482 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-88) SPMAsyncTask::PollTask: Polling task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:56,508 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-88) BaseAsyncTask::OnTaskEndSuccess: Task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:56,510 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-88) CommandAsyncTask::EndActionIfNecessary: All tasks of command 51674dc7-7774-4c78-8bb6-301f046ea07b has ended -> executing endAction >2014-08-21 11:06:56,511 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-88) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 51674dc7-7774-4c78-8bb6-301f046ea07b): calling endAction . >2014-08-21 11:06:56,512 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:06:56,527 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-33) [disks_create_04957bf8-6fd6-4ba3] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:06:56,550 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:06:56,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, imageId = a95ef6a4-d3a2-421f-aa63-0bfaa543fd86), log id: 5cb8d46d >2014-08-21 11:06:56,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@ae15ce5a, log id: 5cb8d46d >2014-08-21 11:06:56,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) Correlation ID: disks_create_04957bf8-6fd6-4ba3, Call Stack: null, Custom Event ID: -1, Message: The disk vm_2_Disk1 was successfully added to VM vm_2. >2014-08-21 11:06:56,609 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:06:56,609 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:06:56,615 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 73c605d7-0794-4766-ad42-b045d5bc24a8 >2014-08-21 11:06:56,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 73c605d7-0794-4766-ad42-b045d5bc24a8), log id: 7161f5fe >2014-08-21 11:06:56,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=73c605d7-0794-4766-ad42-b045d5bc24a8), log id: 565d30d9 >2014-08-21 11:06:56,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMClearTaskVDSCommand, log id: 565d30d9 >2014-08-21 11:06:56,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMClearTaskVDSCommand, log id: 7161f5fe >2014-08-21 11:06:56,705 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) BaseAsyncTask::RemoveTaskFromDB: Removed task 73c605d7-0794-4766-ad42-b045d5bc24a8 from DataBase >2014-08-21 11:06:56,706 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 51674dc7-7774-4c78-8bb6-301f046ea07b >2014-08-21 11:07:04,626 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:07:04,718 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, IsVmDuringInitiatingVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae), log id: 4e1e9295 >2014-08-21 11:07:04,719 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4e1e9295 >2014-08-21 11:07:04,774 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Running command: RunVmCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:07:04,908 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@8540b969), log id: 2fc77a89 >2014-08-21 11:07:04,911 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2fc77a89 >2014-08-21 11:07:04,944 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, vm=VM [vm_2]), log id: 3443d9df >2014-08-21 11:07:05,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, vm=VM [vm_2]), log id: 2a17474d >2014-08-21 11:07:05,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiZmJjMGM0ZjAtMjRmOS00NWNj >LTgwMTEtMGJjYTllMmYyNGI2IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=2aba7272-a8c0-4106-9c5f-11d71d123617}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=188a64c6-c5ad-4f41-8b45-90ba7c9fb221}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=b83e7afd-5557-456a-9d5c-24c0bcaf53cc}, {shared=false, volumeID=a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, index=0, iface=virtio, imageID=e346cbb0-7ed5-4c80-a189-b02706afc39c, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=e346cbb0-7ed5-4c80-a189-b02706afc39c}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=007e6004-52e2-4a7c-896a-e0870bbe4b70}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=4d44beb2-d497-4b81-94ae-3dc402c8ce4f}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=f22ced93-a7cd-47ce-91a2-09f5ba1c5b17}],vmName=vm_2,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:07:05,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2a17474d >2014-08-21 11:07:05,049 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 3443d9df >2014-08-21 11:07:05,050 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:07:05,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Correlation ID: vms_syncAction_5f3e4580-66f2-49d4, Job ID: f00d9c7a-4620-4535-8f3c-73efc583c419, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:07:07,745 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) VM vm_2 61b86c61-6e16-439f-8fcd-486a2448e2ae moved from WaitForLaunch --> PoweringUp >2014-08-21 11:07:07,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[61b86c61-6e16-439f-8fcd-486a2448e2ae]), log id: 41ce4a52 >2014-08-21 11:07:07,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=4860, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@4e716f3e, vmName=vm_2, fileTransferEnable=true, cpuType=Conroe}], log id: 41ce4a52 >2014-08-21 11:07:07,769 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a spice Device without an address when processing VM 61b86c61-6e16-439f-8fcd-486a2448e2ae devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5903, type=graphics} >2014-08-21 11:07:36,205 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) Setting new tasks map. The map contains now 2 tasks >2014-08-21 11:08:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:08:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:10:17,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-58) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 11:10:17,210 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:17,211 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-21 11:10:17,227 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-21 11:10:22,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:22,479 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:22,480 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:22,501 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:27,633 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = Unknown, exit status = Normal), log id: 7e72464a >2014-08-21 11:10:27,646 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SetVmStatusVDSCommand, log id: 7e72464a >2014-08-21 11:10:27,684 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was set to the Unknown status. >2014-08-21 11:10:27,685 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = Unknown, exit status = Normal), log id: 5465096e >2014-08-21 11:10:27,690 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SetVmStatusVDSCommand, log id: 5465096e >2014-08-21 11:10:27,727 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was set to the Unknown status. >2014-08-21 11:10:27,728 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Server failed to respond, vds_id = a8fdd580-c752-4537-9cae-e49e468db7da, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 2, spm_status = None, non-responsive_timeout (seconds) = 61, error = java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,773 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-08-21 11:10:27,791 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-44) ResourceManager::vdsNotResponding entered for Host a8fdd580-c752-4537-9cae-e49e468db7da, rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:27,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:27,870 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:10:28,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:10:28,006 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:28,038 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:10:28,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 73e311b6 >2014-08-21 11:10:28,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 73e311b6 >2014-08-21 11:10:28,636 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:32,889 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:32,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:32,896 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:37,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:37,985 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:38,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:43,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:43,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:43,072 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:48,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) [43495829] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:48,143 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [43495829] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:48,148 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) [43495829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:53,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:53,215 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [70db29ad] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:53,222 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [70db29ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:58,282 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:58,284 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:58,290 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:03,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:03,351 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:03,358 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:08,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:08,455 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:08,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:13,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:13,530 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:13,535 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:18,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:18,599 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:18,604 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:23,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:23,686 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:23,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:28,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:28,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:28,762 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:31,696 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] SSH Soft Fencing command failed on host rose07.qa.lab.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-21 11:11:31,875 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:11:31,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:11:31,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:11:32,009 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:11:32,095 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:11:33,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:33,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:33,881 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:38,993 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:38,994 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:39,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:44,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:44,089 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:44,094 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:49,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:49,156 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:49,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:54,230 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:54,232 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:54,244 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:59,307 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:59,309 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:59,322 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:02,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Correlation ID: 17e68cb, Job ID: a399c5af-2ae7-436e-b214-2c36c691fab2, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-08-21 11:12:02,167 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:02,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:02,268 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:02,317 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:02,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 63793b05 >2014-08-21 11:12:02,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 63793b05 >2014-08-21 11:12:02,576 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:02,582 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:02,613 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 715d0a4e >2014-08-21 11:12:04,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:04,431 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:04,436 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:04,461 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, SetVdsStatusVDSCommand, log id: 715d0a4e >2014-08-21 11:12:04,570 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:04,571 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:04,602 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:04,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d799cca >2014-08-21 11:12:04,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, FenceVdsVDSCommand, return: Done, log id: 3d799cca >2014-08-21 11:12:04,744 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:12:09,745 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:12:09,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:09,823 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:09,857 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:09,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4d08de01 >2014-08-21 11:12:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 4d08de01 >2014-08-21 11:12:10,025 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:12:10,026 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = Down, exit status = Error), log id: 4a44a190 >2014-08-21 11:12:10,033 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, SetVmStatusVDSCommand, log id: 4a44a190 >2014-08-21 11:12:10,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_ha was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:12:10,080 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:12:10,095 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = Down, exit status = Error), log id: 4103c53b >2014-08-21 11:12:10,100 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] FINISH, SetVmStatusVDSCommand, log id: 4103c53b >2014-08-21 11:12:10,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [258c6729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_2 was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:12:10,143 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:12:10,191 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 4752060e >2014-08-21 11:12:10,195 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 4752060e >2014-08-21 11:12:10,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [33e6082a] Correlation ID: 24ef6141, Job ID: 9c3cf8e6-bb89-45a4-b22e-76a435432fb8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-08-21 11:12:10,286 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-44) [5b771b1f] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:10,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:10,443 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:10,451 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:10,482 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b9f5bef >2014-08-21 11:12:10,487 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, SetVdsStatusVDSCommand, log id: 2b9f5bef >2014-08-21 11:12:10,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:10,660 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:10,720 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:10,722 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, IsVmDuringInitiatingVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b), log id: 74565e4d >2014-08-21 11:12:10,723 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 74565e4d >2014-08-21 11:12:10,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4d7ee243 >2014-08-21 11:12:10,743 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] Running command: RunVmCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:12:10,825 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, UpdateVmDynamicDataVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@1bba8144), log id: 39175f0c >2014-08-21 11:12:10,850 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, UpdateVmDynamicDataVDSCommand, log id: 39175f0c >2014-08-21 11:12:10,854 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, CreateVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 39c0b42a >2014-08-21 11:12:10,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, CreateVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 5125dfe0 >2014-08-21 11:12:10,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, FenceVdsVDSCommand, return: Done, log id: 4d7ee243 >2014-08-21 11:12:10,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=4efe0fe5-f86d-4359-a0b8-f6a1cc19feb4}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=080970ae-d7bb-494c-a4fd-832420935d2a}, {shared=false, index=0, volumeID=de7bf293-5790-426c-9f8a-c391b1396c8a, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, imageID=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, specParams={}, optional=false, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, readonly=false, deviceId=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=5db0d96e-73dc-4370-a7fd-9b8f19330e3a}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=c15b2bc7-299d-4376-b060-fcf9ee1bca2c}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=4493feb7-855b-4bb5-9dd3-828324951568}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:12:10,932 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:12:10,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, CreateVDSCommand, log id: 5125dfe0 >2014-08-21 11:12:11,007 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 39c0b42a >2014-08-21 11:12:11,008 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:12:11,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [4256016c] Correlation ID: 4256016c, Job ID: dd25a8fa-dd86-4745-895c-bcf72cfe3a6a, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:12:12,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:12,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:12,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:15,301 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from WaitForLaunch --> PoweringUp >2014-08-21 11:12:15,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-75) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb], vmIds=[04545787-d6d5-4b4d-a06f-2d42db7c933b]), log id: 66a2483f >2014-08-21 11:12:15,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.2.45, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=17176, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@283554bd, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 66a2483f >2014-08-21 11:12:15,319 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Received a spice Device without an address when processing VM 04545787-d6d5-4b4d-a06f-2d42db7c933b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.2.45}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:12:15,933 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:12:15,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:16,000 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:16,031 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:16,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 54f3887e >2014-08-21 11:12:16,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 54f3887e >2014-08-21 11:12:16,200 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:12:16,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: 154fb5a1, Job ID: c8caa1b6-3deb-4496-b3a8-0afb45c17926, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-08-21 11:12:16,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: 17e68cb, Job ID: f4acc64f-c0e3-4e74-8d85-cfe2e10399bd, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-08-21 11:12:16,224 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:17,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:17,691 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:17,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:22,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:22,757 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:22,766 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:27,855 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:27,856 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:27,894 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:32,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:32,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:32,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:38,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:38,085 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:38,098 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:43,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:43,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:43,165 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:48,239 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:48,241 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:48,246 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:53,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:53,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:53,315 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:58,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:58,394 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:58,399 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [799af561] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:03,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:03,460 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:03,466 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:08,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:08,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:08,586 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:12,995 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from PoweringUp --> Up >2014-08-21 11:13:13,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: 4256016c, Job ID: dd25a8fa-dd86-4745-895c-bcf72cfe3a6a, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:13:13,013 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-30) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:13:13,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:13,652 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:13,658 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:18,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:18,718 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:18,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:23,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:23,795 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:23,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:28,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:28,863 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:28,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:33,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:33,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:33,936 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:38,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:38,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:39,007 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:42,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 48f00df7 >2014-08-21 11:13:42,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GetHardwareInfoVDSCommand, log id: 48f00df7 >2014-08-21 11:13:42,682 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [5dcc5fc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:42,690 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [38cc3fee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:13:42,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:13:42,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 30646a5f >2014-08-21 11:13:42,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 30646a5f >2014-08-21 11:13:42,973 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:13:43,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 23690ebf >2014-08-21 11:13:43,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 23690ebf >2014-08-21 11:13:43,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:43,347 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) [60da23ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:13:43,384 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) [60da23ba] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:13:43,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 26908e6d >2014-08-21 11:13:43,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 26908e6d >2014-08-21 11:13:43,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 638d6e0f >2014-08-21 11:13:43,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, SetMOMPolicyParametersVDSCommand, log id: 638d6e0f >2014-08-21 11:13:43,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:13:43,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: 38cc3fee, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:13:43,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [428627dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:45,786 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,787 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,788 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_7432283e-79e5-4f24] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:45,794 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,795 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,851 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:45,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_7432283e-79e5-4f24, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:13:46,185 INFO [org.ovirt.engine.core.bll.StopVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] Running command: StopVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:46,189 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] START, DestroyVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, force=false, secondsToWait=0, gracefully=false, reason=), log id: 374138b8 >2014-08-21 11:13:46,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] START, DestroyVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, force=false, secondsToWait=0, gracefully=false, reason=), log id: 35065f7f >2014-08-21 11:13:46,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] FINISH, DestroyVDSCommand, log id: 35065f7f >2014-08-21 11:13:46,695 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] FINISH, DestroyVmVDSCommand, return: Down, log id: 374138b8 >2014-08-21 11:13:46,703 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-27) [2b4fe416] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:13:46,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] Correlation ID: vms_syncAction_350fd20a-0c01-48b2, Job ID: 565ad661-bfbf-4359-a769-e193a942ad30, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha powered off by admin (Host: monique-vds01.tlv.redhat.com) (Reason: Not Specified). >2014-08-21 11:13:46,981 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:13:46,998 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Running command: RemoveVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:46,999 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = ImageLocked, exit status = Normal), log id: 14bd70ee >2014-08-21 11:13:47,002 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] FINISH, SetVmStatusVDSCommand, log id: 14bd70ee >2014-08-21 11:13:47,004 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:13:47,015 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-3) [25a5ba8e] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:47,019 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:13:47,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, postZeros = false, forceDelete = false), log id: 2db25ab7 >2014-08-21 11:13:47,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] FINISH, DeleteImageGroupVDSCommand, log id: 2db25ab7 >2014-08-21 11:13:47,198 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [5c92cdab] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 71a3406c-1e85-462a-a223-f29c4635425c >2014-08-21 11:13:47,199 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [5c92cdab] CommandMultiAsyncTasks::AttachTask: Attaching task 08748083-1564-4d95-9a52-7c0702981579 to command 71a3406c-1e85-462a-a223-f29c4635425c. >2014-08-21 11:13:47,259 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [5c92cdab] Adding task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:13:47,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5c92cdab] Correlation ID: vms_delete_2628eafd-a5f7-40af, Job ID: eb72cf33-8b2e-401a-9a2a-9fd2716b6aa4, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was successfully removed. >2014-08-21 11:13:47,361 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [5c92cdab] BaseAsyncTask::startPollingTask: Starting to poll task 08748083-1564-4d95-9a52-7c0702981579. >2014-08-21 11:13:47,600 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:13:47,649 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Running command: RemoveVmCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:13:47,650 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = ImageLocked, exit status = Normal), log id: 5c6605d0 >2014-08-21 11:13:47,654 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] FINISH, SetVmStatusVDSCommand, log id: 5c6605d0 >2014-08-21 11:13:47,669 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:13:47,692 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-5) [78764fe1] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:13:47,695 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:13:47,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, postZeros = false, forceDelete = false), log id: 49de24e3 >2014-08-21 11:13:48,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] FINISH, DeleteImageGroupVDSCommand, log id: 49de24e3 >2014-08-21 11:13:48,121 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-5) [1f31faa4] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command e62513d1-c307-402a-9ffe-624682997e88 >2014-08-21 11:13:48,122 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-5) [1f31faa4] CommandMultiAsyncTasks::AttachTask: Attaching task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 to command e62513d1-c307-402a-9ffe-624682997e88. >2014-08-21 11:13:48,133 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-5) [1f31faa4] Adding task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:13:48,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [1f31faa4] Correlation ID: vms_delete_812df52f-aad8-4de7, Job ID: 03fc3aaf-eee1-40cf-97a4-e7f82e159751, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was successfully removed. >2014-08-21 11:13:48,241 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-5) [1f31faa4] BaseAsyncTask::startPollingTask: Starting to poll task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39. >2014-08-21 11:13:48,470 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,471 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,472 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_update_3209115f-f22d-46d6] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:48,527 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,527 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:48,739 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:13:48,772 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:13:48,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5d401652 >2014-08-21 11:13:48,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5d401652 >2014-08-21 11:13:48,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_update_3209115f-f22d-46d6, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:13:56,529 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-08-21 11:13:56,538 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:13:56,544 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) BaseAsyncTask::OnTaskEndSuccess: Task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:13:56,545 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::EndActionIfNecessary: All tasks of command 71a3406c-1e85-462a-a223-f29c4635425c has ended -> executing endAction >2014-08-21 11:13:56,546 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 71a3406c-1e85-462a-a223-f29c4635425c): calling endAction . >2014-08-21 11:13:56,548 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:13:56,549 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:13:56,562 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:13:56,563 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:13:56,564 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:13:56,567 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) BaseAsyncTask::OnTaskEndSuccess: Task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:13:56,569 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::EndActionIfNecessary: All tasks of command e62513d1-c307-402a-9ffe-624682997e88 has ended -> executing endAction >2014-08-21 11:13:56,570 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: e62513d1-c307-402a-9ffe-624682997e88): calling endAction . >2014-08-21 11:13:56,571 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:13:56,583 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 08748083-1564-4d95-9a52-7c0702981579 >2014-08-21 11:13:56,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 08748083-1564-4d95-9a52-7c0702981579), log id: 68931c90 >2014-08-21 11:13:56,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=08748083-1564-4d95-9a52-7c0702981579), log id: 12386793 >2014-08-21 11:13:56,589 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:13:56,591 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:13:56,592 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:13:56,595 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 >2014-08-21 11:13:56,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 48a5c35c-a40f-4e17-80c5-f7b2c740cd39), log id: 6a938c58 >2014-08-21 11:13:56,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] FINISH, HSMClearTaskVDSCommand, log id: 12386793 >2014-08-21 11:13:56,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] FINISH, SPMClearTaskVDSCommand, log id: 68931c90 >2014-08-21 11:13:56,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=48a5c35c-a40f-4e17-80c5-f7b2c740cd39), log id: 6a2e9087 >2014-08-21 11:13:56,625 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] BaseAsyncTask::RemoveTaskFromDB: Removed task 08748083-1564-4d95-9a52-7c0702981579 from DataBase >2014-08-21 11:13:56,626 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 71a3406c-1e85-462a-a223-f29c4635425c >2014-08-21 11:13:56,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] FINISH, HSMClearTaskVDSCommand, log id: 6a2e9087 >2014-08-21 11:13:56,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] FINISH, SPMClearTaskVDSCommand, log id: 6a938c58 >2014-08-21 11:13:56,693 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] BaseAsyncTask::RemoveTaskFromDB: Removed task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 from DataBase >2014-08-21 11:13:56,694 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity e62513d1-c307-402a-9ffe-624682997e88 >2014-08-21 11:15:06,210 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-36) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:15:06,211 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-36) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:15:16,945 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_d5c69dac-42c6-4222] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:17,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:17,076 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:17,077 ERROR [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) VDS status for vds a8fdd580-c752-4537-9cae-e49e468db7da:rose07.qa.lab.tlv.redhat.com is Up >2014-08-21 11:15:17,078 WARN [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) CanDoAction of action StartVds failed. Reasons:VAR__ACTION__START,VDS_STATUS_NOT_VALID_FOR_START >2014-08-21 11:15:17,079 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:17,085 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-3) Operation Failed: [Cannot perform Start operation, Host has to be in one of the following statuses: Down ,Non Responsive or Maintenance.] >2014-08-21 11:15:20,091 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,092 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,093 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_5d901f2e-f132-4c9a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:20,103 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,104 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,128 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:20,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_5d901f2e-f132-4c9a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:15:20,325 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,326 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,327 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_67fc0a0b-2fa5-42c3] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:20,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:20,598 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:20,629 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:20,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 75bcfa45 >2014-08-21 11:15:20,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 75bcfa45 >2014-08-21 11:15:20,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_67fc0a0b-2fa5-42c3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:15:21,413 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:21,444 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 5a7ce233 >2014-08-21 11:15:21,449 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] FINISH, SetVdsStatusVDSCommand, log id: 5a7ce233 >2014-08-21 11:15:21,525 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:21,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Correlation ID: hosts_syncAction_bdf851e1-0229-49a1, Job ID: b08da44b-0cab-459e-8809-b3bd0d09fe35, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-21 11:15:21,828 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:21,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:21,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Removing vds [a8fdd580-c752-4537-9cae-e49e468db7da] from the domain in maintenance cache >2014-08-21 11:15:21,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Removing host(s) [a8fdd580-c752-4537-9cae-e49e468db7da] from hosts unseen domain report cache >2014-08-21 11:15:21,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 2), log id: 5791ac6f >2014-08-21 11:15:24,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, DisconnectStoragePoolVDSCommand, log id: 5791ac6f >2014-08-21 11:15:24,187 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:15:24,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4888b55 >2014-08-21 11:15:24,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4888b55 >2014-08-21 11:15:25,157 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_syncAction_c8fcdef3-2703-4847] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:25,261 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:25,263 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:25,267 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Running command: StopVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:25,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 42c55478 >2014-08-21 11:15:25,302 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, SetVdsStatusVDSCommand, log id: 42c55478 >2014-08-21 11:15:25,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:25,416 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:25,447 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:25,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 190495ec >2014-08-21 11:15:25,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Done, log id: 190495ec >2014-08-21 11:15:25,573 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:15:30,574 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:15:30,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:30,640 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:30,671 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:30,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 67d557a8 >2014-08-21 11:15:30,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 67d557a8 >2014-08-21 11:15:30,805 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:15:30,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_syncAction_c8fcdef3-2703-4847, Job ID: 4e422f17-35fb-4192-9896-fbb25d58c78d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:15:30,813 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:19:15,499 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_122609fa-80a2-4ec3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:19:15,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:15,678 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:15,684 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:19:15,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 16972639 >2014-08-21 11:19:15,721 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: 16972639 >2014-08-21 11:19:15,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:15,818 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:15,849 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:19:15,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5c2a1792 >2014-08-21 11:19:15,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 5c2a1792 >2014-08-21 11:19:15,983 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:19:20,984 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:19:21,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:21,055 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:21,087 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:19:21,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7ea486be >2014-08-21 11:19:21,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7ea486be >2014-08-21 11:19:21,244 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:19:21,275 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_122609fa-80a2-4ec3, Job ID: 2a31d4c9-4922-4ea7-8f60-e102fccbbbaa, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:19:21,280 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:21:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:21:00,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:21:00,709 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:21:04,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 51a19d95 >2014-08-21 11:21:04,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, GetHardwareInfoVDSCommand, log id: 51a19d95 >2014-08-21 11:21:04,562 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [439ab763] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:21:04,571 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [43d2e940] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:21:04,577 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:21:04,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 29c066f5 >2014-08-21 11:21:04,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 29c066f5 >2014-08-21 11:21:04,849 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:21:04,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 5a4350bb >2014-08-21 11:21:05,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, ConnectStoragePoolVDSCommand, log id: 5a4350bb >2014-08-21 11:21:05,168 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:21:05,169 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) [605a093a] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:21:05,200 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) [605a093a] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:21:05,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: ee485f4 >2014-08-21 11:21:05,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: ee485f4 >2014-08-21 11:21:05,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 570fa438 >2014-08-21 11:21:05,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 570fa438 >2014-08-21 11:21:05,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:21:05,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: 43d2e940, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:21:06,081 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [2bdb3d72] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,667 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,668 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,669 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_d25ce43b-f8f3-41a2] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,679 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,679 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,740 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:22,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_d25ce43b-f8f3-41a2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:22:22,899 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,900 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,901 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_19374966-cd46-42d1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,948 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,948 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:23,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:23,199 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:23,229 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:23,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 475d5de2 >2014-08-21 11:22:23,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 475d5de2 >2014-08-21 11:22:23,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_19374966-cd46-42d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:22:26,309 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_3b20c00a-dbe1-4242] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:22:26,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,414 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,419 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:26,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,554 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:26,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2c8bd298 >2014-08-21 11:22:26,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2c8bd298 >2014-08-21 11:22:26,805 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:26,836 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 14acf354 >2014-08-21 11:22:26,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, SetVdsStatusVDSCommand, log id: 14acf354 >2014-08-21 11:22:26,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,938 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,969 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:26,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 77e085ac >2014-08-21 11:22:27,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, FenceVdsVDSCommand, return: Done, log id: 77e085ac >2014-08-21 11:22:27,094 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:22:32,095 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:22:32,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,165 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,202 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:32,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5b0f5c43 >2014-08-21 11:22:32,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 5b0f5c43 >2014-08-21 11:22:32,394 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:22:32,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: 505bca26, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:22:32,473 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [1d0f49a] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:32,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,610 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,627 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:32,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 430da4bb >2014-08-21 11:22:32,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 430da4bb >2014-08-21 11:22:32,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:32,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 71aa9360 >2014-08-21 11:22:32,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 71aa9360 >2014-08-21 11:22:32,982 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:22:37,983 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:22:38,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:38,064 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:38,095 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:38,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5319f5f2 >2014-08-21 11:22:38,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5319f5f2 >2014-08-21 11:22:38,235 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:22:38,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: fd10dc0, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:22:38,244 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_3b20c00a-dbe1-4242, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:22:38,247 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:24:17,450 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:24:17,452 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:24:17,453 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:24:20,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 72064b80 >2014-08-21 11:24:21,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, GetHardwareInfoVDSCommand, log id: 72064b80 >2014-08-21 11:24:21,077 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [6e405299] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:24:21,086 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [2bd2395b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:24:21,092 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:24:21,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d8984d7 >2014-08-21 11:24:21,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: d8984d7 >2014-08-21 11:24:21,396 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:24:21,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 7904b840 >2014-08-21 11:24:21,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, ConnectStoragePoolVDSCommand, log id: 7904b840 >2014-08-21 11:24:21,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:24:21,751 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [6778b85a] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:24:21,782 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [6778b85a] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:24:21,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 72d426cd >2014-08-21 11:24:21,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 72d426cd >2014-08-21 11:24:21,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 63a83ded >2014-08-21 11:24:21,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 63a83ded >2014-08-21 11:24:22,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:24:22,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: 2bd2395b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:24:22,074 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [36ee3909] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,241 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,242 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,243 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_af0cd597-1a75-4519] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,249 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,275 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:39,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_af0cd597-1a75-4519, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:25:39,468 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,468 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,469 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_df081405-3bb6-4c6e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,485 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,485 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:39,693 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:39,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:39,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 1716f2be >2014-08-21 11:25:39,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1716f2be >2014-08-21 11:25:39,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_df081405-3bb6-4c6e, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:25:40,457 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:40,488 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 69be510c >2014-08-21 11:25:40,493 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] FINISH, SetVdsStatusVDSCommand, log id: 69be510c >2014-08-21 11:25:40,566 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:40,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Correlation ID: hosts_syncAction_77934a77-f3b5-409c, Job ID: a470e432-f725-4b83-bb2d-70aa22be7090, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-21 11:25:41,549 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:41,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:41,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing vds [a8fdd580-c752-4537-9cae-e49e468db7da] from the domain in maintenance cache >2014-08-21 11:25:41,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing host(s) [a8fdd580-c752-4537-9cae-e49e468db7da] from hosts unseen domain report cache >2014-08-21 11:25:41,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 2), log id: 72d0406e >2014-08-21 11:25:44,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, DisconnectStoragePoolVDSCommand, log id: 72d0406e >2014-08-21 11:25:44,607 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:25:44,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1fc65a72 >2014-08-21 11:25:44,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 1fc65a72 >2014-08-21 11:25:53,794 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_c7af6e83-3791-4592] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:25:53,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:53,952 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:53,959 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:54,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:54,056 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:54,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:54,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3adc6ca3 >2014-08-21 11:25:54,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3adc6ca3 >2014-08-21 11:25:54,263 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:54,299 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2d02b094 >2014-08-21 11:25:54,304 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, SetVdsStatusVDSCommand, log id: 2d02b094 >2014-08-21 11:25:54,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:54,409 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:54,440 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:54,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 63ed3213 >2014-08-21 11:25:54,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, FenceVdsVDSCommand, return: Done, log id: 63ed3213 >2014-08-21 11:25:54,575 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:25:59,575 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:25:59,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:59,646 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:59,677 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:59,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 372bffb5 >2014-08-21 11:25:59,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 372bffb5 >2014-08-21 11:25:59,807 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:25:59,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: 3a6f926c, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:25:59,869 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [4d99ff20] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:26:00,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:00,072 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:00,076 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:26:00,107 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7139e185 >2014-08-21 11:26:00,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 7139e185 >2014-08-21 11:26:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:00,212 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:00,243 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:26:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 48085c2b >2014-08-21 11:26:00,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 48085c2b >2014-08-21 11:26:00,368 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:26:05,369 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:26:05,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:05,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:05,489 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:26:05,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6bbf1a1c >2014-08-21 11:26:06,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6bbf1a1c >2014-08-21 11:26:06,152 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:26:06,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 6275e16b, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:26:06,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_c7af6e83-3791-4592, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:26:06,184 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:29:07,334 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-21 11:29:07,339 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Running command: ActivateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:07,340 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,341 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,379 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4e8560a9 >2014-08-21 11:29:07,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] FINISH, SetVdsStatusVDSCommand, log id: 4e8560a9 >2014-08-21 11:29:07,426 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_c5d3d199-c3a6-4b84, Job ID: 8c951f28-66b4-4ee6-bcc4-35d63f948c31, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-08-21 11:29:07,452 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-21 11:29:08,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:29:08,233 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:29:08,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:29:08,273 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-08-21 11:29:12,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 1fd96cec >2014-08-21 11:29:12,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GetHardwareInfoVDSCommand, log id: 1fd96cec >2014-08-21 11:29:12,405 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [20fe3322] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:12,413 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [472dd376] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:29:12,419 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:29:12,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 19e8d6d5 >2014-08-21 11:29:12,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 19e8d6d5 >2014-08-21 11:29:12,682 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:29:12,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: b05c29f >2014-08-21 11:29:12,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: b05c29f >2014-08-21 11:29:13,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:13,004 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [ef1a542] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:13,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [ef1a542] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:29:13,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 688e67c2 >2014-08-21 11:29:13,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 688e67c2 >2014-08-21 11:29:13,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 34769bc9 >2014-08-21 11:29:13,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, SetMOMPolicyParametersVDSCommand, log id: 34769bc9 >2014-08-21 11:29:13,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:29:13,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: 472dd376, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:29:13,324 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,065 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,066 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,067 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_c7b5b417-0474-4272] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,080 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,080 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,105 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:18,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_c7b5b417-0474-4272, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:29:18,365 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,366 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,367 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_208f0beb-c867-4bab] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,456 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,456 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:18,853 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:18,912 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:18,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 657bdccc >2014-08-21 11:29:19,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 657bdccc >2014-08-21 11:29:19,120 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:19,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1b3fcabe >2014-08-21 11:29:19,372 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:19,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1b3fcabe >2014-08-21 11:29:19,374 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:29:19,721 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:29:19,721 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:29:19,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:19,845 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:19,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:19,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 154fa521 >2014-08-21 11:29:20,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:20,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 154fa521 >2014-08-21 11:29:20,100 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:20,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_208f0beb-c867-4bab, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:29:23,600 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_syncAction_2dd2b5e3-cb4f-4e77] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:29:23,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:23,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:23,847 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:24,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:24,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:24,114 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:24,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3badc02c >2014-08-21 11:29:24,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3badc02c >2014-08-21 11:29:24,318 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:24,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 723b7e84 >2014-08-21 11:29:24,509 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:24,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 723b7e84 >2014-08-21 11:29:24,511 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:29:24,762 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:29:24,763 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:29:25,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:25,059 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:25,060 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:25,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 573fe75c >2014-08-21 11:29:25,259 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:25,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 573fe75c >2014-08-21 11:29:25,353 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:25,429 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12dd188a >2014-08-21 11:29:25,433 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, SetVdsStatusVDSCommand, log id: 12dd188a >2014-08-21 11:29:25,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:25,650 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:25,746 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:25,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 238265f1 >2014-08-21 11:29:25,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, FenceVdsVDSCommand, return: Done, log id: 238265f1 >2014-08-21 11:29:25,999 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:29:31,000 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:29:31,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:31,127 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:31,187 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:31,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7a000d81 >2014-08-21 11:29:31,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 7a000d81 >2014-08-21 11:29:31,316 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:29:31,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: 38b39414, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:29:31,396 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-8) [7f50e682] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:31,664 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:31,666 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:31,671 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:31,752 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6e693b16 >2014-08-21 11:29:31,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, SetVdsStatusVDSCommand, log id: 6e693b16 >2014-08-21 11:29:32,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:32,010 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:32,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:32,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 53db00da >2014-08-21 11:29:32,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Done, log id: 53db00da >2014-08-21 11:29:32,259 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:29:37,260 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:29:37,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:37,397 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:37,468 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:37,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5043ccef >2014-08-21 11:29:37,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5043ccef >2014-08-21 11:29:37,610 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:29:37,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 12ea24eb, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:29:37,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_syncAction_2dd2b5e3-cb4f-4e77, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:29:37,633 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:31:15,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:31:15,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:31:15,273 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:31:18,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 99cc92e >2014-08-21 11:31:18,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, GetHardwareInfoVDSCommand, log id: 99cc92e >2014-08-21 11:31:19,024 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [e46b1a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:31:19,032 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [7e63a6eb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:31:19,039 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:31:19,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2be9cbf3 >2014-08-21 11:31:19,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 2be9cbf3 >2014-08-21 11:31:19,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:31:19,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6a828d8d >2014-08-21 11:31:19,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 6a828d8d >2014-08-21 11:31:19,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:31:19,786 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:31:19,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:31:19,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 78299c3c >2014-08-21 11:31:20,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 78299c3c >2014-08-21 11:31:20,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 67304916 >2014-08-21 11:31:20,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, SetMOMPolicyParametersVDSCommand, log id: 67304916 >2014-08-21 11:31:20,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:31:20,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: 7e63a6eb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:31:20,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:38,795 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,796 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,797 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_4b70da29-d815-447d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:38,805 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,806 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:38,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_4b70da29-d815-447d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:32:38,995 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,996 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,997 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_236b845d-6a2a-4c00] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:39,012 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,013 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,048 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:39,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_236b845d-6a2a-4c00, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:32:39,409 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,410 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,411 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_accb1d5a-68e3-43c1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:39,437 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,438 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:39,810 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:32:39,868 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:32:39,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 43772bc9 >2014-08-21 11:33:20,072 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:33:20,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 43772bc9 >2014-08-21 11:33:20,075 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:33:20,314 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:33:20,315 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:33:20,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:33:20,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:33:20,459 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:33:20,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 199aa376 >2014-08-21 11:34:00,629 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:00,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 199aa376 >2014-08-21 11:34:00,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_accb1d5a-68e3-43c1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:34:03,540 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_5787e32d-62bf-4865] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:34:03,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:03,730 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:03,744 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:34:03,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:03,931 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:04,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:34:04,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 239a79a1 >2014-08-21 11:34:44,209 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:44,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 239a79a1 >2014-08-21 11:34:44,213 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:34:44,526 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:34:44,527 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:34:44,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:44,653 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:44,654 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:34:44,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4381621a >2014-08-21 11:35:24,928 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:35:24,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 4381621a >2014-08-21 11:35:25,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:35:25,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 26c3dbed >2014-08-21 11:35:25,276 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:35:25,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 26c3dbed >2014-08-21 11:35:25,278 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:35:25,643 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:35:25,644 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:35:25,964 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:25,965 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:25,966 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:35:25,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1234e230 >2014-08-21 11:35:26,159 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:35:26,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1234e230 >2014-08-21 11:35:26,306 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:35:26,409 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 60943cfe >2014-08-21 11:35:26,415 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, SetVdsStatusVDSCommand, log id: 60943cfe >2014-08-21 11:35:26,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:26,603 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:26,661 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:35:26,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: a1f55c9 >2014-08-21 11:35:46,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Done, log id: a1f55c9 >2014-08-21 11:35:46,808 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:35:51,809 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:35:51,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:51,931 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:51,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:35:51,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 48f57bce >2014-08-21 11:36:32,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:36:32,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 48f57bce >2014-08-21 11:36:32,165 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:36:32,407 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:36:32,408 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:36:32,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:36:32,530 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:36:32,531 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:36:32,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 28013745 >2014-08-21 11:37:12,712 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:37:12,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 28013745 >2014-08-21 11:37:12,716 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Host rose07.qa.lab.tlv.redhat.com Primary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-21 11:37:12,776 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-21 11:37:12,778 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-21 11:37:13,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:13,024 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:13,084 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:13,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 4e8d0da6 >2014-08-21 11:37:13,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4e8d0da6 >2014-08-21 11:37:13,245 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:37:18,246 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:37:18,398 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:18,399 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:18,459 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:18,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 2eafb3d2 >2014-08-21 11:37:18,690 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:37:18,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 2eafb3d2 >2014-08-21 11:37:18,692 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:37:18,940 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:37:18,941 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:37:19,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:19,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:19,072 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:19,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 3d0d68db >2014-08-21 11:37:19,287 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:37:19,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 3d0d68db >2014-08-21 11:37:19,289 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Host rose07.qa.lab.tlv.redhat.com Secondary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-21 11:37:19,350 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-21 11:37:19,351 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-21 11:37:19,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to power fence host rose07.qa.lab.tlv.redhat.com. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >2014-08-21 11:37:19,417 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: 4a55168e, Job ID: a0886db3-1810-461b-afa2-71441cdc5bcc, Call Stack: null, Custom Event ID: -1, Message: Failed to stop Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-21 11:37:19,422 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Restart host action failed, updating host a8fdd580-c752-4537-9cae-e49e468db7da to NonResponsive >2014-08-21 11:37:19,483 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7ca3a618 >2014-08-21 11:37:19,487 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, SetVdsStatusVDSCommand, log id: 7ca3a618 >2014-08-21 11:37:19,491 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: hosts_syncAction_5787e32d-62bf-4865, Job ID: a0886db3-1810-461b-afa2-71441cdc5bcc, Call Stack: null, Custom Event ID: -1, Message: Failed to restart Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-21 11:37:19,494 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:37:19,495 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [] >2014-08-21 11:37:20,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 11011807 >2014-08-21 11:37:21,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, GetHardwareInfoVDSCommand, log id: 11011807 >2014-08-21 11:37:21,124 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [2e6c4d38] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:37:21,133 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-79) [7ad48c4f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:37:21,138 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:37:21,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d08d0dd >2014-08-21 11:37:21,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: d08d0dd >2014-08-21 11:37:21,211 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:37:21,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 2131eb5e >2014-08-21 11:37:21,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 2131eb5e >2014-08-21 11:37:21,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:21,641 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:21,700 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:37:21,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: e50d277 >2014-08-21 11:38:01,881 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:01,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: e50d277 >2014-08-21 11:38:01,884 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:38:02,126 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:38:02,127 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:38:02,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:02,251 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:02,252 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:38:02,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45a4dd72 >2014-08-21 11:38:42,452 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:42,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 45a4dd72 >2014-08-21 11:38:42,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: a90f2d5 >2014-08-21 11:38:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, SetMOMPolicyParametersVDSCommand, log id: a90f2d5 >2014-08-21 11:38:42,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:38:42,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:42,697 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: 7ad48c4f, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com power management. >2014-08-21 11:38:42,765 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [351eedf6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,440 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,441 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,442 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_b4c960f5-713e-44fa] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,446 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,447 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,471 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:52,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_b4c960f5-713e-44fa, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:52,659 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,659 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,660 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_239a8a7a-e400-4521] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,685 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,685 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,708 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:52,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_239a8a7a-e400-4521, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:52,988 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,989 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,990 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_a31a3e51-df1e-410a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:53,015 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:53,015 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:53,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:53,222 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:53,254 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:53,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6a723d1d >2014-08-21 11:38:53,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6a723d1d >2014-08-21 11:38:53,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_a31a3e51-df1e-410a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:57,164 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_1c241878-146a-49cc] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:38:57,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:57,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:57,422 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:57,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:57,630 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:57,673 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:57,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4819b274 >2014-08-21 11:38:57,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4819b274 >2014-08-21 11:38:57,994 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:58,036 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 9e1c1d0 >2014-08-21 11:38:58,062 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, SetVdsStatusVDSCommand, log id: 9e1c1d0 >2014-08-21 11:38:58,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:58,325 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:58,356 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:58,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 29347f4d >2014-08-21 11:38:58,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, FenceVdsVDSCommand, return: Done, log id: 29347f4d >2014-08-21 11:38:58,503 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:39:03,504 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:39:03,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:03,572 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:03,602 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:03,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 423725d5 >2014-08-21 11:39:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 423725d5 >2014-08-21 11:39:03,737 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:39:03,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: 1838c425, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:39:03,781 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-7) [411d0e1a] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:39:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:03,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:03,923 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:39:03,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6aa0553d >2014-08-21 11:39:03,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVdsStatusVDSCommand, log id: 6aa0553d >2014-08-21 11:39:04,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:04,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:04,085 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:04,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2c2cac3e >2014-08-21 11:39:04,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Done, log id: 2c2cac3e >2014-08-21 11:39:04,217 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:39:09,218 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:39:09,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:09,292 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:09,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:09,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 47b0b71a >2014-08-21 11:39:09,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 47b0b71a >2014-08-21 11:39:09,472 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:39:09,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 55453975, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:39:09,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_syncAction_1c241878-146a-49cc, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:39:09,512 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:40:47,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:40:47,067 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:40:47,068 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:40:50,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 78823549 >2014-08-21 11:40:50,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] FINISH, GetHardwareInfoVDSCommand, log id: 78823549 >2014-08-21 11:40:50,825 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [598a33c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:40:50,836 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [2f31657c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:40:50,841 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [25697095] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:40:50,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 115261d >2014-08-21 11:40:51,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 115261d >2014-08-21 11:40:51,093 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [25697095] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:40:51,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 74d99b54 >2014-08-21 11:40:51,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 74d99b54 >2014-08-21 11:40:51,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:40:51,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [25697095] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:40:51,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [25697095] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:40:51,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 134f0420 >2014-08-21 11:40:51,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 134f0420 >2014-08-21 11:40:51,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 63851b2d >2014-08-21 11:40:51,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, SetMOMPolicyParametersVDSCommand, log id: 63851b2d >2014-08-21 11:40:51,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:40:51,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: 2f31657c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:40:51,799 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:10,667 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:42:12,839 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,840 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,840 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_e57e2441-5695-496f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:12,844 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,845 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,870 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:12,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_e57e2441-5695-496f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:42:13,096 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,097 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,098 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_9fee5846-4807-4217] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:13,144 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,145 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:13,346 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:13,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:13,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 760ebdec >2014-08-21 11:42:13,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 760ebdec >2014-08-21 11:42:13,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_9fee5846-4807-4217, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:42:16,667 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_b264b449-6c7a-417a] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:42:16,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:16,771 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:16,775 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:16,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:16,883 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:16,913 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:16,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 25343c22 >2014-08-21 11:42:17,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 25343c22 >2014-08-21 11:42:17,085 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:17,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 76b8dcd >2014-08-21 11:42:17,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, SetVdsStatusVDSCommand, log id: 76b8dcd >2014-08-21 11:42:17,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:17,247 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:17,278 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:17,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4ddb8ca2 >2014-08-21 11:42:17,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4ddb8ca2 >2014-08-21 11:42:17,403 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:42:22,404 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:42:22,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,500 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:22,532 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:22,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4a2e75a0 >2014-08-21 11:42:22,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 4a2e75a0 >2014-08-21 11:42:22,662 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:42:22,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: 7834da4e, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:42:22,708 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-3) [55a33e7d] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:22,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:22,848 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:22,882 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 237fbb5d >2014-08-21 11:42:22,886 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, SetVdsStatusVDSCommand, log id: 237fbb5d >2014-08-21 11:42:22,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,980 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:23,011 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:23,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 420a63f7 >2014-08-21 11:42:23,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Done, log id: 420a63f7 >2014-08-21 11:42:23,134 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:42:28,135 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:42:28,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:28,204 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:28,235 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:28,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4333599a >2014-08-21 11:42:28,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4333599a >2014-08-21 11:42:28,366 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:42:28,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: 4ba29363, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:42:28,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_syncAction_b264b449-6c7a-417a, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:42:28,377 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:44:06,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:44:06,238 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:44:06,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:44:09,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 788c62eb >2014-08-21 11:44:09,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetHardwareInfoVDSCommand, log id: 788c62eb >2014-08-21 11:44:09,921 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [21c808bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:44:09,932 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-49) [f746b40] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:44:09,938 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:44:09,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2914d44e >2014-08-21 11:44:10,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 2914d44e >2014-08-21 11:44:10,222 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:44:10,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6528dcdb >2014-08-21 11:44:10,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6528dcdb >2014-08-21 11:44:10,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:44:10,640 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:44:10,682 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:44:10,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 113f7e16 >2014-08-21 11:44:11,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 113f7e16 >2014-08-21 11:44:11,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 3e4865c4 >2014-08-21 11:44:11,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3e4865c4 >2014-08-21 11:44:11,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:44:11,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: f746b40, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:44:11,495 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [54395203] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:28,803 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:45:29,970 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,971 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,972 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_563fb608-6a6c-4c9e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:29,976 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,976 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,003 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:45:30,010 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_563fb608-6a6c-4c9e, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:45:30,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,251 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_d6a998f1-f386-4c34] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:30,285 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,286 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:45:30,486 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-2) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:45:30,517 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-2) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:45:30,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-2) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4dedd0b >2014-08-21 11:45:30,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-2) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4dedd0b >2014-08-21 11:45:30,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_d6a998f1-f386-4c34, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:45:59,658 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 in problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:47:21,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond >2014-08-21 11:47:21,207 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:21,209 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-44) [697a661d] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 11:47:21,248 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [697a661d] Correlation ID: 697a661d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-21 11:47:21,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-08-21 11:47:21,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-76) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 11:47:21,278 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:47:21,279 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:47:21,331 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 11:47:21,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 11:47:21,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-21 11:47:21,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:47:21,401 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 11:47:21,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 11:47:21,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 18f2a9a1 >2014-08-21 11:47:21,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] SpmStopVDSCommand:: vds monique-vds01.tlv.redhat.com is in Connecting status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:47:21,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] FINISH, SpmStopVDSCommand, log id: 18f2a9a1 >2014-08-21 11:47:21,408 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] spm stop on spm failed, stopping spm selection! >2014-08-21 11:47:24,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3d1203f9 >2014-08-21 11:47:24,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:47:24,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-21 11:47:24,430 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is initializing. Message: Recovering from crash or Initializing >2014-08-21 11:47:24,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2014-08-21 11:47:27,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: fbbec82 >2014-08-21 11:47:27,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GetHardwareInfoVDSCommand, log id: fbbec82 >2014-08-21 11:47:27,954 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [e9cdbb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:47:28,000 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [e9cdbb4] Cancelling the recovery from crash timer for VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb because vds started initializing >2014-08-21 11:47:28,002 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [25f17de4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:28,008 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [43630027] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:28,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [43630027] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bfb965 >2014-08-21 11:47:31,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:47:31,505 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 11:47:31,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 11:47:31,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-97) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 6a5634c5 >2014-08-21 11:47:31,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-08-21 11:47:31,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:47:31,589 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-08-21 11:47:31,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, SpmStopVDSCommand, log id: 6a5634c5 >2014-08-21 11:47:31,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) spm stop succeeded, continuing with spm selection >2014-08-21 11:47:31,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId 1, LVER 7 >2014-08-21 11:47:31,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=1, prevLVER=7, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 393a12c1 >2014-08-21 11:47:31,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling started: taskId = ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4 >2014-08-21 11:47:44,765 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:47:55,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling ended: taskId = ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4 task status = finished >2014-08-21 11:47:55,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling ended, spm status: SPM >2014-08-21 11:47:55,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4), log id: 63b593aa >2014-08-21 11:47:55,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, HSMClearTaskVDSCommand, log id: 63b593aa >2014-08-21 11:47:55,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f6cb33c, log id: 393a12c1 >2014-08-21 11:47:55,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 11:47:55,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:47:55,287 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:47:55,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 446f1df1 >2014-08-21 11:47:55,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:47:55,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 69b57d3c >2014-08-21 11:47:55,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69b57d3c >2014-08-21 11:47:55,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 446f1df1 >2014-08-21 11:47:55,384 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-35) Discovered no tasks on Storage Pool DC1 >2014-08-21 11:48:06,877 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-08-21 11:48:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-08-21 11:48:28,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [43630027] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4bfb965 >2014-08-21 11:48:28,086 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [43630027] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 11:48:28,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6e9fdc0f >2014-08-21 11:49:28,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a']] >2014-08-21 11:49:28,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:49:28,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) Command ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a' >2014-08-21 11:49:28,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 6e9fdc0f >2014-08-21 11:49:28,537 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-12) Could not connect host monique-vds01.tlv.redhat.com to pool DC1 with the message: null >2014-08-21 11:49:28,724 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:49:28,727 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 10c185 >2014-08-21 11:49:28,844 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] FINISH, SetVdsStatusVDSCommand, log id: 10c185 >2014-08-21 11:49:28,996 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: e28c63d, Job ID: a469e856-d59f-4b83-8715-08b983d77cf3, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-08-21 11:49:29,109 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 11:49:29,128 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: 25f17de4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host monique-vds01.tlv.redhat.com to Storage Pool DC1 >2014-08-21 11:49:29,141 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [5806b578] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:49:29,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [5806b578] Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-08-21 11:49:40,128 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_35ae1321-07b3-41f2] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:49:40,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:40,336 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:40,341 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:40,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:40,472 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:40,503 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:40,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d0253f6 >2014-08-21 11:49:41,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3d0253f6 >2014-08-21 11:49:41,161 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:41,192 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 45e405f1 >2014-08-21 11:49:41,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 11:49:41,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 15237350 >2014-08-21 11:49:41,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 527aa183 >2014-08-21 11:49:41,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:49:41,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, SpmStopVDSCommand, log id: 527aa183 >2014-08-21 11:49:41,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, ResetIrsVDSCommand, log id: 15237350 >2014-08-21 11:49:41,288 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, SetVdsStatusVDSCommand, log id: 45e405f1 >2014-08-21 11:49:41,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:41,420 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:41,451 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:41,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d7db5ed >2014-08-21 11:49:41,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, FenceVdsVDSCommand, return: Done, log id: 3d7db5ed >2014-08-21 11:49:41,576 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:49:46,108 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : no reporting hosts >2014-08-21 11:49:46,113 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [22a988a8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:49:46,119 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [22a988a8] Correlation ID: 22a988a8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:49:46,577 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:49:46,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:46,674 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:46,706 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:46,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6bc37cdf >2014-08-21 11:49:46,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6bc37cdf >2014-08-21 11:49:46,837 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:49:46,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: 1ef77c35, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:49:46,883 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-7) [4fbd435b] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:47,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:47,073 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:47,077 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:47,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5de3ab47 >2014-08-21 11:49:47,143 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVdsStatusVDSCommand, log id: 5de3ab47 >2014-08-21 11:49:47,272 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:47,273 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:47,352 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:47,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4b6d160a >2014-08-21 11:49:47,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Done, log id: 4b6d160a >2014-08-21 11:49:47,475 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:49:52,476 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:49:52,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:52,625 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:52,662 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:52,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d07d294 >2014-08-21 11:49:52,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3d07d294 >2014-08-21 11:49:52,808 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:49:52,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 6a8e1df6, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:49:52,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_syncAction_35ae1321-07b3-41f2, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:49:52,820 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:50:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering 1 hosts >2014-08-21 11:50:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering hosts id: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, name : monique-vds01.tlv.redhat.com >2014-08-21 11:50:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock Acquired to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS >, sharedLocks= ] >2014-08-21 11:50:00,017 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:50:00,018 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Before acquiring lock in order to prevent monitoring for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,019 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock acquired, from now a monitoring of host will be skipped for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: dcf92b1 >2014-08-21 11:50:00,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] FINISH, SetVdsStatusVDSCommand, log id: dcf92b1 >2014-08-21 11:50:00,093 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Activate finished. Lock released. Monitoring can run now for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Correlation ID: 53b2f72b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was autorecovered. >2014-08-21 11:50:00,098 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS >, sharedLocks= ] >2014-08-21 11:50:02,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 78d23540 >2014-08-21 11:50:02,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 78d23540 >2014-08-21 11:50:02,334 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [78f9a316] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:50:02,382 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [5edecdb3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:50:02,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:50:02,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5bf512c1 >2014-08-21 11:50:59,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 >2014-08-21 11:50:59,750 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-21 11:50:59,779 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [2b61e701] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-21 11:50:59,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [2b61e701] Removing vds [] from the domain in maintenance cache >2014-08-21 11:50:59,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [2b61e701] Removing host(s) [] from hosts unseen domain report cache >2014-08-21 11:50:59,800 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) Finished reconstruct for pool ab2440ae-4286-4465-beed-c2ccc8256f8b. Clearing event queue >2014-08-21 11:51:02,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5bf512c1 >2014-08-21 11:51:02,449 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 11:51:02,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: c8b92d8 >2014-08-21 11:51:32,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:51:32,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:51:32,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:51:36,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 2d562b1a >2014-08-21 11:51:36,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 2d562b1a >2014-08-21 11:51:36,274 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [39c10079] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:51:36,283 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [6adcedc2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:51:36,289 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:51:36,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74522469 >2014-08-21 11:51:36,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 74522469 >2014-08-21 11:51:36,542 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:52:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a']] >2014-08-21 11:52:02,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:52:02,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) Command ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a' >2014-08-21 11:52:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, ConnectStoragePoolVDSCommand, log id: c8b92d8 >2014-08-21 11:52:02,871 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-34) Could not connect host monique-vds01.tlv.redhat.com to pool DC1, as the master domain is in inactive/unknown status - not failing the operation >2014-08-21 11:52:02,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 45009897 >2014-08-21 11:52:02,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 11a8fb51 >2014-08-21 11:52:03,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45009897 >2014-08-21 11:52:03,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [71755e69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-08-21 11:52:03,033 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [71755e69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 11:52:03,044 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [336ed24f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:52:03,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 11a8fb51 >2014-08-21 11:52:03,156 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 in problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:52:03,252 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:52:03,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [5d052fe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:52:03,285 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [5d052fe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:52:03,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4e499d08 >2014-08-21 11:52:03,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4e499d08 >2014-08-21 11:52:03,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 4ce51c58 >2014-08-21 11:52:03,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4ce51c58 >2014-08-21 11:52:03,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:52:03,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: 6adcedc2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:52:03,614 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [656ae0c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:52:06,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-08-21 11:52:06,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 55bfa7be >2014-08-21 11:52:53,517 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:52:56,493 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,494 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,495 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_37ba5b3f-df29-4084] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:52:56,501 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,502 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,530 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:52:56,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_37ba5b3f-df29-4084, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:52:57,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, ConnectStoragePoolVDSCommand, log id: 55bfa7be >2014-08-21 11:52:57,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] hostFromVds::Connected host to pool - selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:52:57,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:52:57,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6eee7651 >2014-08-21 11:52:57,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling started: taskId = d8128341-9a52-4b5b-90e8-5238c9c61cd9 >2014-08-21 11:52:58,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Failed in HSMGetTaskStatusVDS method >2014-08-21 11:52:58,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling ended: taskId = d8128341-9a52-4b5b-90e8-5238c9c61cd9 task status = finished >2014-08-21 11:52:58,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id, code = 661 >2014-08-21 11:52:59,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling ended, spm status: Free >2014-08-21 11:52:59,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=d8128341-9a52-4b5b-90e8-5238c9c61cd9), log id: 50fe5792 >2014-08-21 11:52:59,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, HSMClearTaskVDSCommand, log id: 50fe5792 >2014-08-21 11:52:59,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@593504fc, log id: 6eee7651 >2014-08-21 11:52:59,087 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:52:59,095 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [27da6a25] Correlation ID: 27da6a25, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:52:59,099 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-08-21 11:52:59,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-21 11:52:59,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:52:59,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:52:59,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 794c3b34 >2014-08-21 11:52:59,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling started: taskId = 76a348e2-0af2-456f-b08d-360d146e84d9 >2014-08-21 11:53:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling ended: taskId = 76a348e2-0af2-456f-b08d-360d146e84d9 task status = finished >2014-08-21 11:53:00,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling ended, spm status: SPM >2014-08-21 11:53:00,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=76a348e2-0af2-456f-b08d-360d146e84d9), log id: 626a4429 >2014-08-21 11:53:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] FINISH, HSMClearTaskVDSCommand, log id: 626a4429 >2014-08-21 11:53:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48c1753b, log id: 794c3b34 >2014-08-21 11:53:00,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 11:53:00,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [27da6a25] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:53:00,427 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:53:00,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 3dc4fde6 >2014-08-21 11:53:00,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:53:00,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 1567bc3c >2014-08-21 11:53:00,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1567bc3c >2014-08-21 11:53:00,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dc4fde6 >2014-08-21 11:53:00,588 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-2) [27da6a25] Discovered no tasks on Storage Pool DC1 >2014-08-21 11:53:03,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 recovered from problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:53:03,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-21 12:09:41,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:09:41,897 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:09:41,900 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 12:09:41,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 2b65d4c7 >2014-08-21 12:09:42,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2b65d4c7 >2014-08-21 12:09:48,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:09:48,684 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:09:48,687 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:blablabla.com, User:asfasf, Options: >2014-08-21 12:09:48,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blablabla.com, port = , type = ipmilan, user = asfasf, password = ******, options = ''), log id: 5d2bbda5 >2014-08-21 12:10:27,346 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed] >2014-08-21 12:10:27,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed], log id: 5d2bbda5 >2014-08-21 12:10:27,349 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 12:10:27,360 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host , no running proxy Host was found. >2014-08-21 12:10:27,361 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 12:10:27,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:10:27,372 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:10:27,373 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:blablabla.com, User:asfasf, Options: >2014-08-21 12:10:27,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blablabla.com, port = , type = ipmilan, user = asfasf, password = ******, options = ''), log id: 28301e85 >2014-08-21 12:11:05,589 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed] >2014-08-21 12:11:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed], log id: 28301e85 >2014-08-21 12:11:31,369 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,370 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,371 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [7560924b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:11:31,375 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,376 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,435 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:11:31,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: 7560924b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 12:20:28,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 12:20:28,996 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 12:20:28,997 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 12:20:29,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-08-21 12:20:32,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 42b9f1bf >2014-08-21 12:20:32,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GetHardwareInfoVDSCommand, log id: 42b9f1bf >2014-08-21 12:20:32,133 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [70f9c04d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:20:32,183 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [64dc3cae] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 12:20:32,188 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 12:20:32,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20ad6e18 >2014-08-21 12:20:32,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 20ad6e18 >2014-08-21 12:20:32,245 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 12:20:32,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 60c6b02e >2014-08-21 12:20:32,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 60c6b02e >2014-08-21 12:20:32,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 196aebf7 >2014-08-21 12:20:32,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 196aebf7 >2014-08-21 12:20:32,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [fbb240a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 12:20:32,613 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [fbb240a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:20:32,630 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [174c1abd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:48:06,885 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-08-21 12:48:07,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-08-21 13:48:06,923 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-08-21 13:48:07,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-08-21 14:47:32,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:47:32,636 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 14:47:32,638 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:47:32,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7e5772fe >2014-08-21 14:47:33,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7e5772fe >2014-08-21 14:47:34,783 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,784 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,785 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [4d400bc4] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:47:34,806 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,807 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:35,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:47:35,010 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:47:35,041 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:47:35,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 57d71205 >2014-08-21 14:47:35,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 57d71205 >2014-08-21 14:47:35,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 4d400bc4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 14:48:06,927 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-08-21 14:48:07,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-08-21 14:51:04,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [22a988a8] Command SpmStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 14:51:04,147 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:51:04,149 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 14:51:04,187 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Correlation ID: 1679f6b2, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host rose07.qa.lab.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-21 14:51:04,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-86) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 14:51:04,262 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:04,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-21 14:51:04,276 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-21 14:51:09,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:09,440 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:09,441 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [799af561] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:09,460 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,500 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:14,554 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Server failed to respond, vds_id = a8fdd580-c752-4537-9cae-e49e468db7da, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,592 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-08-21 14:51:14,606 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-1) ResourceManager::vdsNotResponding entered for Host a8fdd580-c752-4537-9cae-e49e468db7da, rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:14,617 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:14,671 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:51:14,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:51:14,800 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:14,830 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:51:14,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3b5347cf >2014-08-21 14:51:14,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3b5347cf >2014-08-21 14:51:14,972 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:19,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:19,688 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:19,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:24,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:24,750 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) [697a661d] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:24,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [697a661d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:29,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:29,811 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:29,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:34,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:34,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:34,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:39,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:39,941 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:39,946 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:45,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:45,012 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:45,017 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:50,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:50,073 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:50,078 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:55,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:55,135 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:55,140 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:00,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:00,196 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:00,200 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:05,255 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:05,256 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:05,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:10,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) [36ee3909] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:10,318 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) [36ee3909] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:10,323 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [36ee3909] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:15,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:15,379 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:15,385 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:17,990 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] SSH Soft Fencing command failed on host rose07.qa.lab.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-21 14:52:18,109 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:18,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:18,214 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:18,218 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:18,295 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:20,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:20,448 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:20,454 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:25,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:25,510 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:25,515 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:30,570 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [336ed24f] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:30,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [336ed24f] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:30,577 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [336ed24f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:35,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:35,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) [3a17e524] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:35,637 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [3a17e524] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:40,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:40,692 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:40,698 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:45,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:45,754 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:45,789 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:48,339 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2453023d] Correlation ID: 2453023d, Job ID: 0dc11331-e4f5-49df-8cd6-bc32aea78fb3, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-08-21 14:52:48,343 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:48,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2453023d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:48,442 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2453023d] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:48,473 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2453023d] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:48,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: cf1678b >2014-08-21 14:52:49,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: cf1678b >2014-08-21 14:52:49,122 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:49,127 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:49,159 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12552137 >2014-08-21 14:52:50,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [5806b578] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:50,849 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [5806b578] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:50,854 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [5806b578] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:50,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 14:52:50,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 113897f >2014-08-21 14:52:50,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 284e18bb >2014-08-21 14:52:50,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] SpmStopVDSCommand:: vds rose07.qa.lab.tlv.redhat.com is in NonResponsive status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:52:50,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SpmStopVDSCommand, log id: 284e18bb >2014-08-21 14:52:50,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, ResetIrsVDSCommand, log id: 113897f >2014-08-21 14:52:50,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SetVdsStatusVDSCommand, log id: 12552137 >2014-08-21 14:52:51,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:51,014 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:51,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 634426cc >2014-08-21 14:52:51,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] SpmStopVDSCommand:: vds rose07.qa.lab.tlv.redhat.com is in Reboot status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:52:51,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SpmStopVDSCommand, log id: 634426cc >2014-08-21 14:52:51,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:51,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 16f4002e >2014-08-21 14:52:51,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, FenceVdsVDSCommand, return: Done, log id: 16f4002e >2014-08-21 14:52:51,271 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 14:52:56,272 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 14:52:56,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,338 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:56,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 33ad64a2 >2014-08-21 14:52:56,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 33ad64a2 >2014-08-21 14:52:56,504 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 14:52:56,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: 6f660a90, Job ID: 79a42b3e-8ff2-4840-9acf-ace2559d63fc, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-08-21 14:52:56,586 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:56,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 629d090 >2014-08-21 14:52:56,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: SocketException: Socket closed >2014-08-21 14:52:56,610 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [1679f6b2] IRS failover failed - cant allocate vds server >2014-08-21 14:52:56,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] FINISH, ResetIrsVDSCommand, log id: 629d090 >2014-08-21 14:52:56,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,757 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:56,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 72f871d4 >2014-08-21 14:52:56,794 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, SetVdsStatusVDSCommand, log id: 72f871d4 >2014-08-21 14:52:56,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,893 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,940 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:56,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4908bf9c >2014-08-21 14:52:57,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, FenceVdsVDSCommand, return: Done, log id: 4908bf9c >2014-08-21 14:52:57,127 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 14:52:58,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:58,976 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:58,981 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:02,127 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 14:53:02,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:53:02,193 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:53:02,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:53:02,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 2b096ecb >2014-08-21 14:53:02,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2b096ecb >2014-08-21 14:53:02,357 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 14:53:02,360 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: 505f5d3c, Job ID: 3757189b-195f-459b-a1f7-cbb5fdd79b36, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-08-21 14:53:02,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: 2453023d, Job ID: fc223561-9382-4d84-9c98-8dae3ae70120, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-08-21 14:53:02,369 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:53:04,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:04,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:04,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:06,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 14:53:06,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId 2, LVER 9 >2014-08-21 14:53:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=2, prevLVER=9, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20a545ba >2014-08-21 14:53:06,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling started: taskId = def043da-6eb2-4b6a-8516-bcb5c0572940 >2014-08-21 14:53:07,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling ended: taskId = def043da-6eb2-4b6a-8516-bcb5c0572940 task status = finished >2014-08-21 14:53:07,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling ended, spm status: SPM >2014-08-21 14:53:07,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=def043da-6eb2-4b6a-8516-bcb5c0572940), log id: 1990638c >2014-08-21 14:53:07,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, HSMClearTaskVDSCommand, log id: 1990638c >2014-08-21 14:53:07,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63fb86f5, log id: 20a545ba >2014-08-21 14:53:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-21 14:53:07,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [3a17e524] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-21 14:53:07,823 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 14:53:07,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 4a97a40 >2014-08-21 14:53:07,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:53:07,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 2b482353 >2014-08-21 14:53:07,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b482353 >2014-08-21 14:53:07,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a97a40 >2014-08-21 14:53:07,908 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) [3a17e524] Discovered no tasks on Storage Pool DC1 >2014-08-21 14:53:09,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:09,113 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:09,118 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:14,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:14,197 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:14,202 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:19,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:19,258 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:19,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:24,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:24,323 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:24,335 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:29,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:29,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:29,415 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:34,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:34,473 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [e83a83] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:34,478 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [e83a83] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:39,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:39,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:39,542 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:44,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:44,617 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:44,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:49,675 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:49,676 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:49,681 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:54,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:54,738 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:54,743 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:59,795 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:59,797 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:59,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:04,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:04,858 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:04,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:09,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:09,925 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:09,930 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:15,007 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) [351eedf6] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:15,009 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) [351eedf6] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:15,015 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) [351eedf6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:20,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:20,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:20,077 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:25,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:25,133 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:25,138 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:30,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:30,194 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:30,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:34,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 47ab6863 >2014-08-21 14:54:34,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GetHardwareInfoVDSCommand, log id: 47ab6863 >2014-08-21 14:54:34,877 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [499d76ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:54:34,886 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [385cd29d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:54:34,891 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:54:34,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6057c658 >2014-08-21 14:54:35,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 6057c658 >2014-08-21 14:54:35,161 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 14:54:35,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6e8e619f >2014-08-21 14:54:35,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 6e8e619f >2014-08-21 14:54:35,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:54:35,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) [d7111e2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:54:35,542 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) [d7111e2] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:54:35,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 23888b6d >2014-08-21 14:54:35,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 23888b6d >2014-08-21 14:54:35,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 38a8537e >2014-08-21 14:54:35,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 38a8537e >2014-08-21 14:54:35,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 14:54:35,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: 385cd29d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 14:54:35,893 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [22fbf874] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 15:46:44,991 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [21d8ccc6] Lock Acquired to object EngineLock [exclusiveLocks= key: blablatesting value: VM_NAME >, sharedLocks= ] >2014-08-21 15:46:45,080 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [21d8ccc6] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 15:46:45,182 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [21d8ccc6] Correlation ID: 21d8ccc6, Job ID: 77e3046b-bd59-4341-9e54-e471d5b62759, Call Stack: null, Custom Event ID: -1, Message: VM blablatesting was created by admin. >2014-08-21 15:46:45,201 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [21d8ccc6] Lock freed to object EngineLock [exclusiveLocks= key: blablatesting value: VM_NAME >, sharedLocks= ] >2014-08-21 15:46:46,279 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-1) [69910da1] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VM >2014-08-21 15:46:58,935 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [498a74bd] Lock Acquired to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM_DISK_BOOT >, sharedLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >] >2014-08-21 15:46:58,957 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [498a74bd] Running command: AddDiskCommand internal: false. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 15:46:59,047 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [1dd00241] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 15:46:59,056 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [1dd00241] Lock freed to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM_DISK_BOOT >, sharedLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >] >2014-08-21 15:46:59,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [1dd00241] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 24b4bb29-4d32-406c-bd93-9219efc9b41c, imageSizeInBytes = 6442450944, volumeFormat = RAW, newImageId = acc83c70-4778-4e2f-8ea6-db095030a59e, newImageDescription = ), log id: 1b0a3a09 >2014-08-21 15:46:59,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [1dd00241] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 15:46:59,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [1dd00241] FINISH, CreateImageVDSCommand, return: acc83c70-4778-4e2f-8ea6-db095030a59e, log id: 1b0a3a09 >2014-08-21 15:46:59,205 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [1dd00241] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command fcd7aec4-f771-4298-9f14-eee64bcae807 >2014-08-21 15:46:59,206 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [1dd00241] CommandMultiAsyncTasks::AttachTask: Attaching task fd137dc6-b124-4dce-9df9-52110c030ebd to command fcd7aec4-f771-4298-9f14-eee64bcae807. >2014-08-21 15:46:59,213 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [1dd00241] Adding task fd137dc6-b124-4dce-9df9-52110c030ebd (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 15:46:59,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [1dd00241] Correlation ID: 498a74bd, Job ID: ab29a0ca-007d-49aa-b82d-450617822d9c, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of blablatesting_Disk1 was initiated on VM blablatesting by admin. >2014-08-21 15:46:59,231 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [1dd00241] BaseAsyncTask::startPollingTask: Starting to poll task fd137dc6-b124-4dce-9df9-52110c030ebd. >2014-08-21 15:47:07,163 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-68) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-08-21 15:47:07,169 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-68) SPMAsyncTask::PollTask: Polling task fd137dc6-b124-4dce-9df9-52110c030ebd (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 15:47:07,174 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-68) BaseAsyncTask::OnTaskEndSuccess: Task fd137dc6-b124-4dce-9df9-52110c030ebd (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 15:47:07,176 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-68) CommandAsyncTask::EndActionIfNecessary: All tasks of command fcd7aec4-f771-4298-9f14-eee64bcae807 has ended -> executing endAction >2014-08-21 15:47:07,176 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-68) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: fcd7aec4-f771-4298-9f14-eee64bcae807): calling endAction . >2014-08-21 15:47:07,177 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-48) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 15:47:07,202 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-48) [498a74bd] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 15:47:07,220 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-48) [1dd00241] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 15:47:07,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [1dd00241] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 24b4bb29-4d32-406c-bd93-9219efc9b41c, imageId = acc83c70-4778-4e2f-8ea6-db095030a59e), log id: 4faf1dfb >2014-08-21 15:47:07,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [1dd00241] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@a06b6e64, log id: 4faf1dfb >2014-08-21 15:47:07,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) Correlation ID: 498a74bd, Call Stack: null, Custom Event ID: -1, Message: The disk blablatesting_Disk1 was successfully added to VM blablatesting. >2014-08-21 15:47:07,296 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 15:47:07,297 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 15:47:07,301 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-48) SPMAsyncTask::ClearAsyncTask: Attempting to clear task fd137dc6-b124-4dce-9df9-52110c030ebd >2014-08-21 15:47:07,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = fd137dc6-b124-4dce-9df9-52110c030ebd), log id: 1ae468f7 >2014-08-21 15:47:07,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=fd137dc6-b124-4dce-9df9-52110c030ebd), log id: 9a73dec >2014-08-21 15:47:07,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, HSMClearTaskVDSCommand, log id: 9a73dec >2014-08-21 15:47:07,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, SPMClearTaskVDSCommand, log id: 1ae468f7 >2014-08-21 15:47:07,355 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-48) BaseAsyncTask::RemoveTaskFromDB: Removed task fd137dc6-b124-4dce-9df9-52110c030ebd from DataBase >2014-08-21 15:47:07,356 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity fcd7aec4-f771-4298-9f14-eee64bcae807 >2014-08-21 15:47:08,561 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-2) [64110950] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 15:47:08,612 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-2) [35cb29b4] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VM >2014-08-21 15:47:08,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [35cb29b4] Correlation ID: 35cb29b4, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM blablatesting. (User: admin) >2014-08-21 15:47:08,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [35cb29b4] Correlation ID: 64110950, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM blablatesting. (User: admin) >2014-08-21 15:47:19,737 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-3) [6345e02e] Lock Acquired to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >, sharedLocks= ] >2014-08-21 15:47:19,795 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-3) [6345e02e] START, IsVmDuringInitiatingVDSCommand( vmId = ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b), log id: 6bfd2d73 >2014-08-21 15:47:19,796 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-3) [6345e02e] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6bfd2d73 >2014-08-21 15:47:19,871 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] Running command: RunVmCommand internal: false. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VM >2014-08-21 15:47:19,956 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-8-thread-33) [6345e02e] Started HA reservation scoring method >2014-08-21 15:47:20,124 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@856b2b88), log id: 3d02a987 >2014-08-21 15:47:20,127 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] FINISH, UpdateVmDynamicDataVDSCommand, log id: 3d02a987 >2014-08-21 15:47:20,161 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, vm=VM [blablatesting]), log id: 34953aca >2014-08-21 15:47:20,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] START, CreateVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, vm=VM [blablatesting]), log id: 665bf461 >2014-08-21 15:47:20,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=f443002c-e9af-4639-a970-6134f67e03fb}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=70d489d9-b4c5-4b8c-bc30-ae4ad0f28347}, {shared=false, index=0, volumeID=acc83c70-4778-4e2f-8ea6-db095030a59e, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, imageID=24b4bb29-4d32-406c-bd93-9219efc9b41c, specParams={}, optional=false, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, readonly=false, deviceId=24b4bb29-4d32-406c-bd93-9219efc9b41c}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=16a8a1b9-ae03-4d42-a9d5-ca94f2c781d2}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=3c2663c3-18f6-4a14-893d-7d6654d2c73b}],vmName=blablatesting,cpuType=Conroe,fileTransferEnable=true >2014-08-21 15:47:20,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] FINISH, CreateVDSCommand, log id: 665bf461 >2014-08-21 15:47:20,292 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 34953aca >2014-08-21 15:47:20,293 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-33) [6345e02e] Lock freed to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >, sharedLocks= ] >2014-08-21 15:47:20,297 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [6345e02e] Correlation ID: 6345e02e, Job ID: 1978a54a-a1cc-420b-97d5-940aa0080b41, Call Stack: null, Custom Event ID: -1, Message: VM blablatesting was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 15:47:21,965 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) VM blablatesting ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b moved from WaitForLaunch --> PoweringUp >2014-08-21 15:47:21,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-92) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b]), log id: 44a02334 >2014-08-21 15:47:21,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-92) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=5908, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@6d674e1e, vmName=blablatesting, fileTransferEnable=true, cpuType=Conroe}], log id: 44a02334 >2014-08-21 15:47:21,987 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Received a memballoon Device without an address when processing VM ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-08-21 15:47:21,988 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Received a spice Device without an address when processing VM ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 15:48:06,936 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-08-21 15:48:07,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-21 15:48:07,468 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] Lock Acquired to object EngineLock [exclusiveLocks= key: fb6a24f7-251c-44ff-9fa1-eff823539f45 value: DISK >key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227 value: DISK >key: 59645b6f-d70b-45fb-aa66-c582c373d58a value: STORAGE >, sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: OVF_UPDATE >] >2014-08-21 15:48:07,471 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 15:48:07,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] START, SetVolumeDescriptionVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 8d3bfe3a-2e8c-4857-9fd8-979894bfa227, imageId = 54d3176e-454f-447e-adb6-8e69a0d998df), log id: ecb03d2 >2014-08-21 15:48:07,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-21 15:48:07,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] -- setVolumeDescription parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGroupGUID=8d3bfe3a-2e8c-4857-9fd8-979894bfa227 > volUUID=54d3176e-454f-447e-adb6-8e69a0d998df > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"59645b6f-d70b-45fb-aa66-c582c373d58a"}],"Last Updated":"Sun Aug 17 13:47:06 IDT 2014"} > >2014-08-21 15:48:07,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [65ece037] FINISH, SetVolumeDescriptionVDSCommand, log id: ecb03d2 >2014-08-21 15:48:07,696 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] Running command: UploadStreamCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 15:48:07,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] START, UploadStreamVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: dbd227d >2014-08-21 15:48:07,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- executeVdsBrokerCommand: >2014-08-21 15:48:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGUID=8d3bfe3a-2e8c-4857-9fd8-979894bfa227 > volUUID=54d3176e-454f-447e-adb6-8e69a0d998df > size=10,240 > >2014-08-21 15:48:08,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] FINISH, UploadStreamVDSCommand, log id: dbd227d >2014-08-21 15:48:08,015 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-52) [51750b49] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 68f4fcb1-c68c-4efe-8597-7d0531c248b9 >2014-08-21 15:48:08,016 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-52) [51750b49] CommandMultiAsyncTasks::AttachTask: Attaching task 32b2abc4-d8b8-4638-a619-9f6be786ff3b to command 68f4fcb1-c68c-4efe-8597-7d0531c248b9. >2014-08-21 15:48:08,062 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-52) [51750b49] Adding task 32b2abc4-d8b8-4638-a619-9f6be786ff3b (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 15:48:08,077 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] Lock freed to object EngineLock [exclusiveLocks= key: fb6a24f7-251c-44ff-9fa1-eff823539f45 value: DISK >key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227 value: DISK >key: 59645b6f-d70b-45fb-aa66-c582c373d58a value: STORAGE >, sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: OVF_UPDATE >] >2014-08-21 15:48:08,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] START, SetVolumeDescriptionVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 8d3bfe3a-2e8c-4857-9fd8-979894bfa227, imageId = 54d3176e-454f-447e-adb6-8e69a0d998df), log id: 7ebb25c3 >2014-08-21 15:48:08,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-21 15:48:08,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- setVolumeDescription parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGroupGUID=8d3bfe3a-2e8c-4857-9fd8-979894bfa227 > volUUID=54d3176e-454f-447e-adb6-8e69a0d998df > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"59645b6f-d70b-45fb-aa66-c582c373d58a"}],"Last Updated":"Thu Aug 21 15:48:07 IDT 2014","Size":10240} > >2014-08-21 15:48:08,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] FINISH, SetVolumeDescriptionVDSCommand, log id: 7ebb25c3 >2014-08-21 15:48:08,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] START, SetVolumeDescriptionVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = fb6a24f7-251c-44ff-9fa1-eff823539f45, imageId = 852ec5d1-f14b-4d80-bb15-0b6c1d375ec8), log id: 2b7863c9 >2014-08-21 15:48:08,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-21 15:48:08,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] -- setVolumeDescription parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGroupGUID=fb6a24f7-251c-44ff-9fa1-eff823539f45 > volUUID=852ec5d1-f14b-4d80-bb15-0b6c1d375ec8 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"59645b6f-d70b-45fb-aa66-c582c373d58a"}],"Last Updated":"Sun Aug 17 13:47:06 IDT 2014"} > >2014-08-21 15:48:08,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [51750b49] FINISH, SetVolumeDescriptionVDSCommand, log id: 2b7863c9 >2014-08-21 15:48:08,350 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Running command: UploadStreamCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 15:48:08,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] START, UploadStreamVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 464669d3 >2014-08-21 15:48:08,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] -- executeVdsBrokerCommand: >2014-08-21 15:48:08,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] -- parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGUID=fb6a24f7-251c-44ff-9fa1-eff823539f45 > volUUID=852ec5d1-f14b-4d80-bb15-0b6c1d375ec8 > size=10,240 > >2014-08-21 15:48:09,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] FINISH, UploadStreamVDSCommand, log id: 464669d3 >2014-08-21 15:48:09,643 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-52) [1fe6377c] CommandMultiAsyncTasks::AttachTask: Attaching task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 to command 68f4fcb1-c68c-4efe-8597-7d0531c248b9. >2014-08-21 15:48:09,650 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Adding task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 15:48:09,653 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Trying to release a shared lock for key: ab2440ae-4286-4465-beed-c2ccc8256f8bOVF_UPDATE , but lock does not exist >2014-08-21 15:48:09,654 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Trying to release exclusive lock which does not exist, lock key: fb6a24f7-251c-44ff-9fa1-eff823539f45DISK >2014-08-21 15:48:09,655 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Trying to release exclusive lock which does not exist, lock key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227DISK >2014-08-21 15:48:09,655 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Trying to release exclusive lock which does not exist, lock key: 59645b6f-d70b-45fb-aa66-c582c373d58aSTORAGE >2014-08-21 15:48:09,656 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] Lock freed to object EngineLock [exclusiveLocks= key: fb6a24f7-251c-44ff-9fa1-eff823539f45 value: DISK >key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227 value: DISK >key: 59645b6f-d70b-45fb-aa66-c582c373d58a value: STORAGE >, sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: OVF_UPDATE >] >2014-08-21 15:48:09,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] START, SetVolumeDescriptionVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = fb6a24f7-251c-44ff-9fa1-eff823539f45, imageId = 852ec5d1-f14b-4d80-bb15-0b6c1d375ec8), log id: 3f6e1a47 >2014-08-21 15:48:09,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-21 15:48:09,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] -- setVolumeDescription parameters: > spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b > sdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a > imageGroupGUID=fb6a24f7-251c-44ff-9fa1-eff823539f45 > volUUID=852ec5d1-f14b-4d80-bb15-0b6c1d375ec8 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"59645b6f-d70b-45fb-aa66-c582c373d58a"}],"Last Updated":"Thu Aug 21 15:48:07 IDT 2014","Size":10240} > >2014-08-21 15:48:09,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-52) [1fe6377c] FINISH, SetVolumeDescriptionVDSCommand, log id: 3f6e1a47 >2014-08-21 15:48:09,775 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) [1fe6377c] BaseAsyncTask::startPollingTask: Starting to poll task 32b2abc4-d8b8-4638-a619-9f6be786ff3b. >2014-08-21 15:48:09,776 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) [1fe6377c] BaseAsyncTask::startPollingTask: Starting to poll task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32. >2014-08-21 15:48:17,178 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-14) Polling and updating Async Tasks: 3 tasks, 2 tasks to poll now >2014-08-21 15:48:17,186 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-14) SPMAsyncTask::PollTask: Polling task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 15:48:17,187 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-14) BaseAsyncTask::OnTaskEndSuccess: Task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 15:48:17,189 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-14) Task with DB Task ID c4fd4537-85a6-4d7a-bfe7-2f0fb3867616 and VDSM Task ID 32b2abc4-d8b8-4638-a619-9f6be786ff3b is in state Polling. End action for command 68f4fcb1-c68c-4efe-8597-7d0531c248b9 will proceed when all the entitys tasks are completed. >2014-08-21 15:48:17,190 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-14) SPMAsyncTask::PollTask: Polling task 32b2abc4-d8b8-4638-a619-9f6be786ff3b (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 15:48:17,191 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-14) BaseAsyncTask::OnTaskEndSuccess: Task 32b2abc4-d8b8-4638-a619-9f6be786ff3b (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 15:48:17,192 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-14) CommandAsyncTask::EndActionIfNecessary: All tasks of command 68f4fcb1-c68c-4efe-8597-7d0531c248b9 has ended -> executing endAction >2014-08-21 15:48:17,193 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-14) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 68f4fcb1-c68c-4efe-8597-7d0531c248b9): calling endAction . >2014-08-21 15:48:17,194 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-1) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-08-21 15:48:17,202 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-08-21 15:48:17,203 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-1) [65ece037] Trying to release a shared lock for key: ab2440ae-4286-4465-beed-c2ccc8256f8bOVF_UPDATE , but lock does not exist >2014-08-21 15:48:17,204 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-1) [65ece037] Trying to release exclusive lock which does not exist, lock key: fb6a24f7-251c-44ff-9fa1-eff823539f45DISK >2014-08-21 15:48:17,205 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-1) [65ece037] Trying to release exclusive lock which does not exist, lock key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227DISK >2014-08-21 15:48:17,205 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-1) [65ece037] Trying to release exclusive lock which does not exist, lock key: 59645b6f-d70b-45fb-aa66-c582c373d58aSTORAGE >2014-08-21 15:48:17,206 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] Lock freed to object EngineLock [exclusiveLocks= key: fb6a24f7-251c-44ff-9fa1-eff823539f45 value: DISK >key: 8d3bfe3a-2e8c-4857-9fd8-979894bfa227 value: DISK >key: 59645b6f-d70b-45fb-aa66-c582c373d58a value: STORAGE >, sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: OVF_UPDATE >] >2014-08-21 15:48:17,207 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-08-21 15:48:17,208 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-08-21 15:48:17,209 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 32b2abc4-d8b8-4638-a619-9f6be786ff3b >2014-08-21 15:48:17,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 32b2abc4-d8b8-4638-a619-9f6be786ff3b), log id: 5994e3b2 >2014-08-21 15:48:17,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=32b2abc4-d8b8-4638-a619-9f6be786ff3b), log id: 401ed07d >2014-08-21 15:48:17,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] FINISH, HSMClearTaskVDSCommand, log id: 401ed07d >2014-08-21 15:48:17,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] FINISH, SPMClearTaskVDSCommand, log id: 5994e3b2 >2014-08-21 15:48:17,284 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] BaseAsyncTask::RemoveTaskFromDB: Removed task 32b2abc4-d8b8-4638-a619-9f6be786ff3b from DataBase >2014-08-21 15:48:17,285 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 >2014-08-21 15:48:17,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 8dca66a6-eb47-4796-8e1c-f47a20dd8c32), log id: 649bdb7b >2014-08-21 15:48:17,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=8dca66a6-eb47-4796-8e1c-f47a20dd8c32), log id: 34669bd7 >2014-08-21 15:48:17,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] FINISH, HSMClearTaskVDSCommand, log id: 34669bd7 >2014-08-21 15:48:17,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-1) [65ece037] FINISH, SPMClearTaskVDSCommand, log id: 649bdb7b >2014-08-21 15:48:17,362 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] BaseAsyncTask::RemoveTaskFromDB: Removed task 8dca66a6-eb47-4796-8e1c-f47a20dd8c32 from DataBase >2014-08-21 15:48:17,363 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-1) [65ece037] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 68f4fcb1-c68c-4efe-8597-7d0531c248b9 >2014-08-21 15:48:23,201 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) VM blablatesting ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b moved from PoweringUp --> Up >2014-08-21 15:48:23,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: 6345e02e, Job ID: 1978a54a-a1cc-420b-97d5-940aa0080b41, Call Stack: null, Custom Event ID: -1, Message: VM blablatesting started on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 15:48:23,304 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-34) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 15:48:36,481 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-26) Setting new tasks map. The map contains now 2 tasks >2014-08-21 15:49:36,482 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Setting new tasks map. The map contains now 0 tasks >2014-08-21 15:49:36,483 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 16:48:06,943 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-08-21 16:48:09,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-08-21 17:21:59,942 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-21 17:21:59,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 17:22:00,020 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [35b8c005] Running command: LogoutBySessionCommand internal: false. >2014-08-21 17:22:00,083 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [2f04c863] Running command: LogoutUserCommand internal: false. >2014-08-21 17:22:00,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [2f04c863] Correlation ID: 2f04c863, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-21 17:48:06,950 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-21 17:48:09,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Attempting to update VMs/Templates Ovf. >2014-08-21 18:48:06,956 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-36) Principal admin::internal synchronized >2014-08-21 18:48:09,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [6f5fe43e] Attempting to update VMs/Templates Ovf. >2014-08-21 19:48:06,960 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-21 19:48:09,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-08-21 20:48:06,972 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-41) Principal admin::internal synchronized >2014-08-21 20:48:09,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [697a661d] Attempting to update VMs/Templates Ovf. >2014-08-21 21:48:07,006 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-91) [43495829] Principal admin::internal synchronized >2014-08-21 21:48:09,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-08-21 22:48:07,012 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-21 22:48:09,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [3a17e524] Attempting to update VMs/Templates Ovf. >2014-08-21 23:48:07,018 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-08-21 23:48:09,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-08-22 00:48:07,023 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-08-22 00:48:09,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-22 01:48:07,028 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-08-22 01:48:09,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-08-22 02:48:07,040 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized >2014-08-22 02:48:09,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-22 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-63) Start deleteAgedOutCommandEntities >2014-08-22 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-71) Start deleteAgedOutAuditLogs >2014-08-22 03:35:35,018 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-63) Finished deleteAgedOutCommandEntities >2014-08-22 03:35:35,047 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-71) Finished deleteAgedOutAuditLogs >2014-08-22 03:48:07,045 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-22 03:48:09,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-08-22 04:48:07,055 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-22 04:48:09,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-08-22 05:48:07,060 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-08-22 05:48:10,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-08-22 06:48:07,068 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-08-22 06:48:10,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-08-22 07:48:07,073 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-91) [43495829] Principal admin::internal synchronized >2014-08-22 07:48:10,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-08-22 08:48:07,077 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-22 08:48:10,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-08-22 09:48:07,083 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-08-22 09:48:10,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-08-22 10:48:07,098 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-13) Principal admin::internal synchronized >2014-08-22 10:48:10,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-08-22 11:48:07,105 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-41) Principal admin::internal synchronized >2014-08-22 11:48:10,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-08-22 12:48:07,116 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-08-22 12:48:10,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) [174c1abd] Attempting to update VMs/Templates Ovf. >2014-08-22 13:48:07,130 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-9) [656ae0c2] Principal admin::internal synchronized >2014-08-22 13:48:10,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-08-22 14:48:07,137 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-36) Principal admin::internal synchronized >2014-08-22 14:48:10,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-08-22 15:48:07,153 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-08-22 15:48:10,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-08-22 16:48:07,161 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-08-22 16:48:10,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-08-22 17:48:07,169 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-22 17:48:10,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Attempting to update VMs/Templates Ovf. >2014-08-22 18:48:07,176 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-08-22 18:48:10,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-08-22 19:48:07,183 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-08-22 19:48:10,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [656ae0c2] Attempting to update VMs/Templates Ovf. >2014-08-22 20:48:07,187 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-88) Principal admin::internal synchronized >2014-08-22 20:48:10,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-08-22 21:48:07,191 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) Principal admin::internal synchronized >2014-08-22 21:48:10,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-08-22 22:48:07,198 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-22 22:48:10,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-22 23:48:07,203 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-08-22 23:48:10,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [2bdb3d72] Attempting to update VMs/Templates Ovf. >2014-08-23 00:48:07,209 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [27da6a25] Principal admin::internal synchronized >2014-08-23 00:48:10,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [43495829] Attempting to update VMs/Templates Ovf. >2014-08-23 01:48:07,214 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-08-23 01:48:10,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-08-23 02:48:07,217 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-91) [43495829] Principal admin::internal synchronized >2014-08-23 02:48:10,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-08-23 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-13) Start deleteAgedOutAuditLogs >2014-08-23 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Start deleteAgedOutCommandEntities >2014-08-23 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-13) Finished deleteAgedOutAuditLogs >2014-08-23 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Finished deleteAgedOutCommandEntities >2014-08-23 03:48:07,225 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-08-23 03:48:10,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-08-23 04:48:07,230 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-08-23 04:48:10,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-08-23 05:48:07,234 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Principal admin::internal synchronized >2014-08-23 05:48:10,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) [174c1abd] Attempting to update VMs/Templates Ovf. >2014-08-23 06:48:07,242 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) [36ee3909] Principal admin::internal synchronized >2014-08-23 06:48:10,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-08-23 07:48:07,249 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-08-23 07:48:10,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-08-23 08:48:07,255 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-39) Principal admin::internal synchronized >2014-08-23 08:48:10,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) [799af561] Attempting to update VMs/Templates Ovf. >2014-08-23 09:48:07,259 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-08-23 09:48:10,580 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [6102b316] Attempting to update VMs/Templates Ovf. >2014-08-23 10:48:07,282 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-23 10:48:10,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [697a661d] Attempting to update VMs/Templates Ovf. >2014-08-23 11:48:07,285 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-36) Principal admin::internal synchronized >2014-08-23 11:48:10,627 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-08-23 12:48:07,290 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) [799af561] Principal admin::internal synchronized >2014-08-23 12:48:10,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-08-23 13:48:07,297 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-23 13:48:10,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-08-23 14:48:07,302 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized >2014-08-23 14:48:10,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [2bdb3d72] Attempting to update VMs/Templates Ovf. >2014-08-23 15:48:07,308 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-08-23 15:48:10,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-08-23 16:48:07,312 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-08-23 16:48:10,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-23 17:48:07,317 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) [22fbf874] Principal admin::internal synchronized >2014-08-23 17:48:10,721 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-08-23 18:48:07,322 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-23 18:48:10,730 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-08-23 19:48:07,330 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-88) Principal admin::internal synchronized >2014-08-23 19:48:10,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-23 20:48:07,336 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-39) Principal admin::internal synchronized >2014-08-23 20:48:10,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-08-23 21:48:07,348 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-08-23 21:48:10,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [6102b316] Attempting to update VMs/Templates Ovf. >2014-08-23 22:48:07,354 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) [174c1abd] Principal admin::internal synchronized >2014-08-23 22:48:10,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Attempting to update VMs/Templates Ovf. >2014-08-23 23:48:07,364 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-08-23 23:48:10,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-24 00:48:07,367 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-08-24 00:48:10,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-08-24 01:48:07,378 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) Principal admin::internal synchronized >2014-08-24 01:48:10,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-24 02:48:07,385 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-24 02:48:10,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) [54395203] Attempting to update VMs/Templates Ovf. >2014-08-24 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-4) [799af561] Start deleteAgedOutCommandEntities >2014-08-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Start deleteAgedOutAuditLogs >2014-08-24 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Finished deleteAgedOutAuditLogs >2014-08-24 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-4) [799af561] Finished deleteAgedOutCommandEntities >2014-08-24 03:48:07,389 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Principal admin::internal synchronized >2014-08-24 03:48:10,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-08-24 04:48:07,392 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-32) Principal admin::internal synchronized >2014-08-24 04:48:10,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-08-24 05:48:07,408 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Principal admin::internal synchronized >2014-08-24 05:48:10,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-08-24 06:48:07,415 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-81) [e83a83] Principal admin::internal synchronized >2014-08-24 06:48:10,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [656ae0c2] Attempting to update VMs/Templates Ovf. >2014-08-24 07:48:07,419 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-24 07:48:10,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) [336ed24f] Attempting to update VMs/Templates Ovf. >2014-08-24 08:48:07,424 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-39) Principal admin::internal synchronized >2014-08-24 08:48:10,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-08-24 09:48:07,431 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-100) Principal admin::internal synchronized >2014-08-24 09:48:10,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-08-24 10:48:07,436 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-08-24 10:48:10,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-24 11:48:07,442 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-08-24 11:48:11,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-08-24 12:48:07,450 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) Principal admin::internal synchronized >2014-08-24 12:48:11,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-08-24 13:48:07,454 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-08-24 13:48:11,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-24 14:48:07,461 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-08-24 14:48:11,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [36ee3909] Attempting to update VMs/Templates Ovf. >2014-08-24 15:48:07,469 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-08-24 15:48:11,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-08-24 16:48:07,473 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-24 16:48:11,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-08-24 17:48:07,479 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) [3a17e524] Principal admin::internal synchronized >2014-08-24 17:48:11,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-08-24 18:48:07,483 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) [174c1abd] Principal admin::internal synchronized >2014-08-24 18:48:11,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-08-24 19:48:07,487 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-93) [5806b578] Principal admin::internal synchronized >2014-08-24 19:48:11,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-08-24 20:48:07,493 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-08-24 20:48:11,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [22fbf874] Attempting to update VMs/Templates Ovf. >2014-08-24 21:48:07,500 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [27da6a25] Principal admin::internal synchronized >2014-08-24 21:48:11,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-08-24 22:48:07,506 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Principal admin::internal synchronized >2014-08-24 22:48:11,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) [5806b578] Attempting to update VMs/Templates Ovf. >2014-08-24 23:48:07,510 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-08-24 23:48:11,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-08-25 00:48:07,514 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Principal admin::internal synchronized >2014-08-25 00:48:11,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-08-25 01:48:07,519 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) [336ed24f] Principal admin::internal synchronized >2014-08-25 01:48:11,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-08-25 02:48:07,526 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-08-25 02:48:11,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-25 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Start deleteAgedOutAuditLogs >2014-08-25 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-68) Start deleteAgedOutCommandEntities >2014-08-25 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Finished deleteAgedOutAuditLogs >2014-08-25 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-68) Finished deleteAgedOutCommandEntities >2014-08-25 03:48:07,530 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Principal admin::internal synchronized >2014-08-25 03:48:11,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Attempting to update VMs/Templates Ovf. >2014-08-25 04:48:07,538 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-25 04:48:11,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) [428627dc] Attempting to update VMs/Templates Ovf. >2014-08-25 05:48:07,543 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-08-25 05:48:11,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Attempting to update VMs/Templates Ovf. >2014-08-25 06:48:07,551 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-08-25 06:48:11,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-08-25 07:48:07,566 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized >2014-08-25 07:48:11,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-08-25 08:48:07,570 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-25 08:48:11,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [2bdb3d72] Attempting to update VMs/Templates Ovf. >2014-08-25 09:48:07,606 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-08-25 09:48:11,395 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) [174c1abd] Attempting to update VMs/Templates Ovf. >2014-08-25 10:48:07,610 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-08-25 10:48:11,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-08-25 11:48:07,614 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-08-25 11:48:11,415 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-08-25 12:48:07,619 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized >2014-08-25 12:48:11,425 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-08-25 13:48:07,625 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-08-25 13:48:11,435 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-08-25 14:10:04,256 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-25 14:10:04,258 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-25 14:10:04,259 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-25 14:10:04,260 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-25 14:10:04,261 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-25 14:10:04,261 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-25 14:10:04,263 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-25 14:10:04,263 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-25 14:10:04,264 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-25 14:10:04,265 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-25 14:10:04,265 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-25 14:10:04,266 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-25 14:10:04,267 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-25 14:10:04,287 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-25 14:10:04,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-25 14:10:04,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-25 14:10:04,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-25 14:10:04,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-25 14:10:04,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-25 14:10:04,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-25 14:10:04,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-25 14:10:04,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-25 14:10:04,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-25 14:10:04,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-25 14:10:04,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-25 14:10:04,296 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-25 14:10:04,296 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-25 14:10:04,297 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-25 14:10:04,298 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-25 14:10:04,310 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-25 14:10:04,311 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-25 14:10:04,312 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-25 14:10:04,312 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-25 14:10:04,313 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-25 14:10:04,314 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-25 14:10:04,314 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-25 14:10:04,315 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-25 14:10:04,316 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-25 14:10:04,317 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-25 14:10:04,317 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-25 14:10:04,318 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-25 14:10:04,319 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-25 14:10:04,319 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-25 14:10:04,320 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-25 14:10:04,321 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-25 14:10:04,341 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-25 14:10:04,342 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-25 14:10:04,343 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-25 14:10:04,344 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-25 14:10:04,344 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-25 14:10:04,345 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-25 14:10:04,346 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-25 14:10:04,347 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-25 14:10:04,347 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-25 14:10:04,348 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-25 14:10:04,349 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-25 14:10:04,349 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-25 14:10:04,350 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-25 14:10:04,350 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-25 14:10:04,351 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-25 14:10:04,352 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-25 14:10:04,365 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-25 14:10:04,365 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-25 14:10:04,366 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-25 14:10:04,367 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-25 14:10:04,367 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-25 14:10:04,368 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-25 14:10:04,369 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-25 14:10:05,936 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-25 14:10:07,131 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-25 14:10:07,150 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-25 14:10:07,398 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140821064931.gitb794d66.el6 >2014-08-25 14:10:07,399 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-25 14:10:07,404 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-25 14:10:07,405 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-25 14:10:07,406 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-25 14:10:07,478 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-25 14:10:07,479 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-25 14:10:07,480 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-25 14:10:07,482 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-25 14:10:07,485 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-25 14:10:07,519 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-25 14:10:07,554 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-25 14:10:07,555 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-25 14:10:07,577 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-25 14:10:07,832 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-25 14:10:07,919 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-25 14:10:07,964 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-25 14:10:07,966 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-25 14:10:07,967 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-25 14:10:07,983 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-25 14:10:07,984 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-25 14:10:08,112 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-25 14:10:08,113 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-25 14:10:08,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-25 14:10:08,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-25 14:10:08,120 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-25 14:10:08,126 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-25 14:10:08,280 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-25 14:10:08,280 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 14:10:08,282 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 14:10:08,283 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-25 14:10:08,624 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-25 14:10:08,626 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-25 14:10:08,659 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-25 14:10:08,661 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-25 14:10:08,899 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 14:10:08,979 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-25 14:10:09,325 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-25 14:10:09,405 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-25 14:10:09,406 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 14:10:09,446 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-25 14:10:09,470 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-25 14:10:09,471 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 14:10:09,499 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-25 14:10:09,529 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-25 14:10:09,594 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-25 14:10:09,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-25 14:10:09,618 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-25 14:10:09,619 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-25 14:10:09,619 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-25 14:10:09,620 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-25 14:10:09,642 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-25 14:10:09,644 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-25 14:10:09,668 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-25 14:10:09,678 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-25 14:10:09,725 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-25 14:10:09,784 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-25 14:10:09,791 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-25 14:10:09,792 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-25 14:10:09,794 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-25 14:10:14,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 3a21bd28 >2014-08-25 14:10:14,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 65effd1a >2014-08-25 14:10:14,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 3a21bd28 >2014-08-25 14:10:14,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 65effd1a >2014-08-25 14:10:15,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 1544ac8f >2014-08-25 14:10:15,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 1544ac8f >2014-08-25 14:10:15,898 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-25 14:10:15,914 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-25 14:10:15,898 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-25 14:10:16,449 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [2a52d990] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-25 14:10:16,488 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [68dd90e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 14:10:16,490 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [1b993ab6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-25 14:10:20,026 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 14:10:20,125 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-25 14:10:20,223 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-25 14:10:20,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: UNASSIGNED not exist in string table >2014-08-25 14:10:20,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-25 14:10:20,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-25 14:10:20,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-25 14:10:20,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-25 14:10:20,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-25 14:10:20,272 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-25 14:10:20,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-25 14:10:20,275 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-25 14:10:20,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-25 14:10:20,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: DWH_STOPPED not exist in string table >2014-08-25 14:10:20,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: DWH_STARTED not exist in string table >2014-08-25 14:10:20,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: DWH_ERROR not exist in string table >2014-08-25 14:10:20,327 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-25 14:10:20,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-25 14:10:20,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-25 14:10:20,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-25 14:10:21,203 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Correlation ID: 2cf3503d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-25 14:10:21,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [2cf3503d] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-25 14:10:21,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-25 14:10:21,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-25 14:10:21,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 7bc810dd >2014-08-25 14:10:22,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-25 14:10:22,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 1448d603 >2014-08-25 14:10:22,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1448d603 >2014-08-25 14:10:22,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bc810dd >2014-08-25 14:10:22,101 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) [2cf3503d] Discovered no tasks on Storage Pool DC1 >2014-08-25 14:10:27,680 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginAdminUserCommand internal: false. >2014-08-25 14:10:27,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 14:10:37,320 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-25 14:10:37,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 14:24:27,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:24:27,794 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 14:24:27,830 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:powet_wait=10, Fencing policy:null >2014-08-25 14:24:27,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'powet_wait=10', policy = 'null'), log id: 29aa499 >2014-08-25 14:24:28,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 29aa499 >2014-08-25 14:36:31,139 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (ajp--127.0.0.1-8702-4) [72b543d5] Lock Acquired to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >, sharedLocks= ] >2014-08-25 14:36:31,595 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-8-thread-44) [72b543d5] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b Type: VMAction group MIGRATE_VM with role type USER >2014-08-25 14:36:31,889 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-44) [72b543d5] START, MigrateVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, srcHost=rose07.qa.lab.tlv.redhat.com, dstVdsId=9e129c9c-c9d3-42a5-a28e-b0c47c89affb, dstHost=monique-vds01.tlv.redhat.com:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 2563cd6 >2014-08-25 14:36:31,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-44) [72b543d5] START, MigrateBrokerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, srcHost=rose07.qa.lab.tlv.redhat.com, dstVdsId=9e129c9c-c9d3-42a5-a28e-b0c47c89affb, dstHost=monique-vds01.tlv.redhat.com:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 6fd8fb2d >2014-08-25 14:36:31,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-44) [72b543d5] FINISH, MigrateBrokerVDSCommand, log id: 6fd8fb2d >2014-08-25 14:36:32,003 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-44) [72b543d5] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 2563cd6 >2014-08-25 14:36:32,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [72b543d5] Correlation ID: 72b543d5, Job ID: 1ebeefd0-abd0-49a3-a92d-9ef79dee7a7f, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: blablatesting, Source: rose07.qa.lab.tlv.redhat.com, Destination: monique-vds01.tlv.redhat.com, User: admin). >2014-08-25 14:36:34,134 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) RefreshVmList vm id ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b is migrating to vds monique-vds01.tlv.redhat.com ignoring it in the refresh until migration is done >2014-08-25 14:36:37,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-41) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb], vmIds=[ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b]), log id: 5b8a2636 >2014-08-25 14:36:37,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-41) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, afterMigrationStatus=, keyboardLayout=en-us, displayIp=10.35.2.45, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=29273, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@20033505, vmName=blablatesting, fileTransferEnable=true, cpuType=Conroe}], log id: 5b8a2636 >2014-08-25 14:36:37,384 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Received a memballoon Device without an address when processing VM ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b devices, skipping device: {alias=balloon0, specParams={model=none}, target=1048576, device=memballoon, type=balloon} >2014-08-25 14:36:37,396 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Received a spice Device without an address when processing VM ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.2.45}, device=spice, tlsPort=5901, type=graphics} >2014-08-25 14:36:37,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: 72b543d5, Job ID: 1ebeefd0-abd0-49a3-a92d-9ef79dee7a7f, Call Stack: null, Custom Event ID: -1, Message: Migration completed (VM: blablatesting, Source: rose07.qa.lab.tlv.redhat.com, Destination: monique-vds01.tlv.redhat.com, Duration: 5 seconds). >2014-08-25 14:36:37,717 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (DefaultQuartzScheduler_Worker-41) Lock freed to object EngineLock [exclusiveLocks= key: ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b value: VM >, sharedLocks= ] >2014-08-25 14:36:37,719 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-7) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-25 14:36:39,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-40) START, DestroyVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b, force=false, secondsToWait=0, gracefully=false, reason=), log id: 5408968c >2014-08-25 14:36:39,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, DestroyVDSCommand, log id: 5408968c >2014-08-25 14:36:39,383 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) RefreshVmList vm id ba5dc9a3-bbef-4f92-ab12-aa8ff06a668b status = Down on vds rose07.qa.lab.tlv.redhat.com ignoring it in the refresh until migration is done >2014-08-25 14:36:47,538 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:36:47,540 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 14:36:47,600 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:36:47,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 3f617924 >2014-08-25 14:36:48,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3f617924 >2014-08-25 14:36:49,393 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 14:36:49,394 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 14:36:49,396 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [61ee3fe4] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-08-25 14:36:49,663 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 14:36:49,664 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 14:36:50,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:36:50,446 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:36:50,501 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:36:50,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 2ac70b10 >2014-08-25 14:36:50,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2ac70b10 >2014-08-25 14:36:50,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: 61ee3fe4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-25 14:36:54,575 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) [7167ef83] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 14:36:54,769 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:36:54,770 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:36:54,807 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-18) [7167ef83] Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 14:36:55,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [7167ef83] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:36:55,094 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [7167ef83] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:36:55,126 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [7167ef83] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:36:55,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [7167ef83] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 392512d0 >2014-08-25 14:36:55,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [7167ef83] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 392512d0 >2014-08-25 14:36:55,591 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 14:36:55,671 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 90fb789 >2014-08-25 14:36:55,712 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] FINISH, SetVdsStatusVDSCommand, log id: 90fb789 >2014-08-25 14:36:55,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:36:55,826 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:36:55,858 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:36:55,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 3b74c04c >2014-08-25 14:36:56,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] FINISH, FenceVdsVDSCommand, return: Test Succeeded, null, log id: 3b74c04c >2014-08-25 14:36:56,098 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-25 14:37:01,119 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-25 14:37:01,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:37:01,190 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:37:01,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:37:01,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1b662970 >2014-08-25 14:37:01,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 1b662970 >2014-08-25 14:37:01,456 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-18) [3c45bede] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-25 14:37:01,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [3c45bede] Correlation ID: 3c45bede, Job ID: 835b7c55-8832-4dfc-a34b-49fec00b2cdb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-25 14:37:01,580 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-18) [7d139dcc] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 14:37:01,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:37:01,760 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:37:01,768 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-18) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 14:37:01,800 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4cfbd9cb >2014-08-25 14:37:01,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, SetVdsStatusVDSCommand, log id: 4cfbd9cb >2014-08-25 14:37:02,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:37:02,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:37:02,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:37:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 4c23ede4 >2014-08-25 14:37:02,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, FenceVdsVDSCommand, return: Test Succeeded, null, log id: 4c23ede4 >2014-08-25 14:37:02,295 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-18) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-25 14:37:06,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:06,716 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:06,717 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:07,296 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-18) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-25 14:37:07,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:37:07,430 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:37:07,461 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:37:07,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1b3e8e77 >2014-08-25 14:37:07,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1b3e8e77 >2014-08-25 14:37:07,705 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-18) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-25 14:37:07,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: 3272272a, Job ID: 835b7c55-8832-4dfc-a34b-49fec00b2cdb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-25 14:37:07,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: 7167ef83, Job ID: 835b7c55-8832-4dfc-a34b-49fec00b2cdb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-25 14:37:07,738 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-18) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 14:37:11,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:11,788 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:11,789 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:16,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:16,866 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:16,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:21,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:21,961 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:21,962 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:27,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:27,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:27,030 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:32,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2a52d990] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:32,096 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [2a52d990] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:32,097 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [2a52d990] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:37,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:37,183 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:37,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:42,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:42,247 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:42,249 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:47,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:47,325 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:47,326 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:52,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:52,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:52,391 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:37:57,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:37:57,461 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:37:57,467 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:02,526 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:02,528 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:02,586 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:07,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2a52d990] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:07,660 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [2a52d990] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:07,672 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [2a52d990] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:12,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:12,734 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:12,746 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:17,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:17,807 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:17,815 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:22,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:22,913 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:22,931 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:28,016 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:28,018 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:28,035 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:33,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [68dd90e1] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 14:38:33,103 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [68dd90e1] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:33,110 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [68dd90e1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:37,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-25 14:38:37,214 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 14:38:37,225 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 14:38:40,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 3ed526ce >2014-08-25 14:38:40,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 3ed526ce >2014-08-25 14:38:41,049 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [3de11e6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 14:38:41,092 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [2a01135c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 14:38:41,132 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 14:38:41,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f8cc7a8 >2014-08-25 14:38:41,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4f8cc7a8 >2014-08-25 14:38:41,483 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-25 14:38:41,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 2c4d583f >2014-08-25 14:38:41,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) FINISH, ConnectStoragePoolVDSCommand, log id: 2c4d583f >2014-08-25 14:38:42,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:38:42,057 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 14:38:42,111 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 14:38:42,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 3dc19ccf >2014-08-25 14:38:42,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3dc19ccf >2014-08-25 14:38:42,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 59c7d828 >2014-08-25 14:38:42,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] FINISH, SetMOMPolicyParametersVDSCommand, log id: 59c7d828 >2014-08-25 14:38:42,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-25 14:38:42,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [34d6a6e0] Correlation ID: 2a01135c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-25 14:38:42,683 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [409e4904] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 14:40:46,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:40:46,646 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 14:40:46,678 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=10, Fencing policy:null >2014-08-25 14:40:46,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=10', policy = 'null'), log id: fa9ce8a >2014-08-25 14:40:46,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: fa9ce8a >2014-08-25 14:40:59,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:40:59,583 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 14:40:59,630 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:sadasas, Fencing policy:null >2014-08-25 14:40:59,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'sadasas', policy = 'null'), log id: 56563d96 >2014-08-25 14:40:59,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 56563d96 >2014-08-25 14:41:38,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 14:41:38,352 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 14:41:38,387 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:-T 10, Fencing policy:null >2014-08-25 14:41:38,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '-T 10', policy = 'null'), log id: 219f88b3 >2014-08-25 14:41:38,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 219f88b3 >2014-08-25 15:04:11,240 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-27) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-25 15:04:11,242 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:04:11,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-25 15:04:11,261 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-25 15:04:16,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:04:16,463 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:04:16,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:04:16,508 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-25 15:04:21,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:04:21,552 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:04:21,621 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Server failed to respond, vds_id = a8fdd580-c752-4537-9cae-e49e468db7da, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.SocketTimeoutException: connect timed out >2014-08-25 15:04:21,666 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-08-25 15:04:21,683 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-23) ResourceManager::vdsNotResponding entered for Host a8fdd580-c752-4537-9cae-e49e468db7da, rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:21,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:04:21,754 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [281ca882] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:04:21,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [281ca882] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:04:21,887 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [281ca882] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:21,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [281ca882] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 15:04:21,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [281ca882] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 46504d69 >2014-08-25 15:04:22,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [281ca882] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 46504d69 >2014-08-25 15:04:22,216 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) [281ca882] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5b0d5fd3 >2014-08-25 15:04:22,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) [281ca882] FINISH, SetVdsStatusVDSCommand, log id: 5b0d5fd3 >2014-08-25 15:04:22,274 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:04:22,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:04:22,380 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:22,385 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-23) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:04:22,521 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:04:52,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Correlation ID: 6840af3b, Job ID: a03b4b4f-2416-43cf-8a27-5c3e2b4c4a36, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-08-25 15:04:52,595 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:04:52,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:04:52,696 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:52,730 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6840af3b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-08-25 15:04:52,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 75307648 >2014-08-25 15:04:53,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 75307648 >2014-08-25 15:04:53,134 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 527cf0bd >2014-08-25 15:04:53,138 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6840af3b] FINISH, SetVdsStatusVDSCommand, log id: 527cf0bd >2014-08-25 15:04:53,218 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-23) [226a0527] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:04:53,223 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-23) [226a0527] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:04:53,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:04:53,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:53,440 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:04:53,473 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 69175502 >2014-08-25 15:04:53,477 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, SetVdsStatusVDSCommand, log id: 69175502 >2014-08-25 15:04:53,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:04:53,579 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:04:53,611 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 } >2014-08-25 15:04:53,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }'), log id: 38bf74f0 >2014-08-25 15:04:53,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Failed in FenceVdsVDS method >2014-08-25 15:04:53,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Command org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand return value > FenceStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=16, mMessage=Unexpected exception]] >2014-08-25 15:04:53,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) HostName = monique-vds01.tlv.redhat.com >2014-08-25 15:04:53,777 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Command FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = '{ fencingEnabled=true, skipFencingIfSDActive=false, skipFencingIfConnectivityBroken=false, hostsWithBrokenConnectivityThreshold=50 }') execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to FenceVdsVDS, error = Unexpected exception, code = 16 >2014-08-25 15:04:53,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, FenceVdsVDSCommand, log id: 38bf74f0 >2014-08-25 15:04:53,782 ERROR [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) Failed to start VDS using Primary Power Management agent >2014-08-25 15:04:53,820 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-08-25 15:04:53,855 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to power fence host rose07.qa.lab.tlv.redhat.com. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >2014-08-25 15:04:53,857 ERROR [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) Command org.ovirt.engine.core.bll.StartVdsCommand throw Vdc Bll exception. With error message VdcBLLException: VDS_FENCE_OPERATION_FAILED (Failed with error VDS_FENCE_OPERATION_FAILED and code 5021) >2014-08-25 15:04:53,864 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: 7ec5b047, Job ID: 0aaacf85-0f2c-42cd-a5a5-bf52a1e597f0, Call Stack: null, Custom Event ID: -1, Message: Failed to start Host rose07.qa.lab.tlv.redhat.com, (User: Engine). >2014-08-25 15:04:53,867 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-23) Restart host action failed, updating host a8fdd580-c752-4537-9cae-e49e468db7da to NonResponsive >2014-08-25 15:04:53,900 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: ab9e0f4 >2014-08-25 15:04:53,935 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, SetVdsStatusVDSCommand, log id: ab9e0f4 >2014-08-25 15:04:53,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: 6840af3b, Job ID: 843e7b05-280c-46a9-b133-ad03ec801005, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-08-25 15:04:53,948 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-23) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:04:57,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:04:57,112 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:04:57,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:05:02,184 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:05:02,186 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:05:02,211 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:05:07,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:05:07,362 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:05:07,419 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [2cf3503d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:05:12,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:05:12,488 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:05:12,495 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:05:17,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:05:17,555 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:05:17,652 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:05:17,789 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-44) [630ee61] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:05:17,985 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-44) [630ee61] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:05:18,053 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [630ee61] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1e008e25 >2014-08-25 15:05:18,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [630ee61] FINISH, SetVdsStatusVDSCommand, log id: 1e008e25 >2014-08-25 15:05:18,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [630ee61] Correlation ID: 630ee61, Job ID: cab02d6d-69ff-4ea4-bcee-aef2b8821a27, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-25 15:05:35,436 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:05:35,453 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:05:35,454 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [5733facb] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-08-25 15:05:35,505 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:05:35,506 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:05:35,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5733facb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:05:35,939 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [5733facb] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:05:35,970 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [5733facb] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:05:35,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [5733facb] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 140be26b >2014-08-25 15:05:36,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [5733facb] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 140be26b >2014-08-25 15:05:36,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5733facb] Correlation ID: 5733facb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-25 15:05:39,627 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) [6fe991c0] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:05:39,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:05:39,899 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:05:39,916 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Running command: StartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:05:40,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:05:40,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:05:40,149 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:05:40,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 5a958ab8 >2014-08-25 15:05:40,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, null, log id: 5a958ab8 >2014-08-25 15:05:40,351 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-25 15:05:45,352 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-25 15:05:45,424 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:05:45,425 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:05:45,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:05:45,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 5b485354 >2014-08-25 15:05:45,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5b485354 >2014-08-25 15:05:45,689 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-25 15:05:45,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Correlation ID: 6fe991c0, Job ID: 625ef8cd-1be0-4b12-a441-f435f690a71a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-25 15:05:45,696 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-23) [6fe991c0] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:08:56,887 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-9) [22339709] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-25 15:08:56,960 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-44) [22339709] Running command: ActivateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:08:56,962 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-44) [22339709] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-25 15:08:56,963 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-44) [22339709] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-25 15:08:57,068 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [22339709] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 32ad35d5 >2014-08-25 15:08:57,105 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [22339709] FINISH, SetVdsStatusVDSCommand, log id: 32ad35d5 >2014-08-25 15:08:57,200 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-44) Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-25 15:08:57,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: 22339709, Job ID: b580a067-ace7-4234-82b8-09904b3fe149, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-08-25 15:08:57,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-44) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-25 15:08:59,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [1b993ab6] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 28c62f30 >2014-08-25 15:08:59,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [1b993ab6] FINISH, GetHardwareInfoVDSCommand, log id: 28c62f30 >2014-08-25 15:08:59,862 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [2580878f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:08:59,872 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [1102379a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:08:59,884 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:08:59,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6a9ecb0a >2014-08-25 15:09:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 6a9ecb0a >2014-08-25 15:09:00,171 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-25 15:09:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 7c55959e >2014-08-25 15:09:00,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7c55959e >2014-08-25 15:09:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [47145cc7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:09:00,706 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [47145cc7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:09:00,738 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [47145cc7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:09:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 5553f0b4 >2014-08-25 15:09:01,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5553f0b4 >2014-08-25 15:09:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 3d21cf40 >2014-08-25 15:09:01,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [47145cc7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3d21cf40 >2014-08-25 15:09:01,137 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [47145cc7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-25 15:09:01,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [47145cc7] Correlation ID: 1102379a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-25 15:09:01,269 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [720b10c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:09:13,342 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-2) [19ba8c62] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:09:13,377 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-2) [19ba8c62] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 231e3b1e >2014-08-25 15:09:13,382 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-2) [19ba8c62] FINISH, SetVdsStatusVDSCommand, log id: 231e3b1e >2014-08-25 15:09:13,653 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-2) [19ba8c62] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:09:13,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [19ba8c62] Correlation ID: 19ba8c62, Job ID: ecdf7001-e2e8-4d0b-8e3e-baef6ade86da, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-25 15:09:13,677 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com >2014-08-25 15:09:13,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:09:13,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Removing vds [a8fdd580-c752-4537-9cae-e49e468db7da] from the domain in maintenance cache >2014-08-25 15:09:13,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Removing host(s) [a8fdd580-c752-4537-9cae-e49e468db7da] from hosts unseen domain report cache >2014-08-25 15:09:13,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 2), log id: 40ed8bd6 >2014-08-25 15:09:17,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, DisconnectStoragePoolVDSCommand, log id: 40ed8bd6 >2014-08-25 15:09:17,437 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [6d426868] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:09:17,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [6d426868] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 91478b7 >2014-08-25 15:09:17,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [6d426868] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 91478b7 >2014-08-25 15:09:19,697 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-1) [79a7668e] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:09:19,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:09:19,847 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:09:19,861 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Running command: StopVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:09:19,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 595923d7 >2014-08-25 15:09:19,897 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] FINISH, SetVdsStatusVDSCommand, log id: 595923d7 >2014-08-25 15:09:20,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:09:20,102 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:09:20,134 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:09:20,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 6074418f >2014-08-25 15:09:20,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, null, log id: 6074418f >2014-08-25 15:09:20,333 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-25 15:09:45,334 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-25 15:09:45,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:09:45,420 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:09:45,452 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:09:45,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 6aa991d6 >2014-08-25 15:09:45,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6aa991d6 >2014-08-25 15:09:45,684 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-25 15:09:45,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Correlation ID: 79a7668e, Job ID: 5c221712-0ec2-4edb-9763-fd5e9a454a42, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-25 15:09:45,694 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-12) [79a7668e] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:10:09,388 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-25 15:10:09,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-08-25 15:10:17,388 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-10) [6d4de647] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:10:17,437 ERROR [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-10) Failed to run StartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-08-25 15:10:17,438 WARN [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-10) CanDoAction of action StartVds failed. Reasons:VAR__ACTION__START,VDS_FENCE_DISABLED_AT_QUIET_TIME,$seconds 148 >2014-08-25 15:10:17,439 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-10) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:14:31,612 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-10) [1cd31788] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:14:31,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:14:31,761 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:14:31,787 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Running command: StartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-08-25 15:14:31,820 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a4df984 >2014-08-25 15:14:31,825 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] FINISH, SetVdsStatusVDSCommand, log id: 7a4df984 >2014-08-25 15:14:32,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:14:32,034 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:14:32,066 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:14:32,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 79ae8f0b >2014-08-25 15:14:32,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] FINISH, FenceVdsVDSCommand, return: Test Succeeded, null, log id: 79ae8f0b >2014-08-25 15:14:32,860 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-25 15:14:35,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:14:35,900 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:14:35,901 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:14:37,862 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-25 15:14:37,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:14:37,935 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:14:38,018 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:14:38,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 6c32ca52 >2014-08-25 15:14:38,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6c32ca52 >2014-08-25 15:14:38,252 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-25 15:14:38,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Correlation ID: 1cd31788, Job ID: a995f291-f870-4932-802b-5d083da1eb78, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-25 15:14:38,261 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-24) [1cd31788] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-25 15:14:40,960 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:14:40,962 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:14:40,963 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:14:46,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:14:46,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:14:46,055 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:14:51,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [6d426868] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:14:51,117 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [6d426868] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:14:51,122 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [6d426868] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:14:56,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:14:56,185 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:14:56,193 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:01,253 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:01,255 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:01,271 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:06,338 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [720b10c7] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:06,340 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [720b10c7] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:06,346 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [720b10c7] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:11,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:11,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:11,420 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:16,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:16,489 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:16,501 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:21,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:21,570 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:21,576 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:26,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:26,666 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:26,672 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:31,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:31,735 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:32,267 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:37,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2a52d990] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:37,348 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [2a52d990] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:37,354 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [2a52d990] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:42,418 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:42,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:42,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:47,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:47,489 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:47,495 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:52,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:52,567 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:52,591 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:15:57,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:15:57,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:15:57,664 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:16:02,725 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-25 15:16:02,727 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:16:02,736 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:16:05,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-25 15:16:05,805 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-25 15:16:05,810 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-25 15:16:09,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6f8bc5ff >2014-08-25 15:16:09,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, GetHardwareInfoVDSCommand, log id: 6f8bc5ff >2014-08-25 15:16:09,538 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [2bef6560] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:16:09,552 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-28) [1c912a72] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:16:09,563 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:16:09,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 47fa6c48 >2014-08-25 15:16:09,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 47fa6c48 >2014-08-25 15:16:09,825 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-25 15:16:09,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6a1c1d5c >2014-08-25 15:16:10,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 6a1c1d5c >2014-08-25 15:16:10,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [b86371b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:16:10,142 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) [b86371b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:16:10,174 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) [b86371b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:16:10,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 1a76b4a7 >2014-08-25 15:16:10,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1a76b4a7 >2014-08-25 15:16:10,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 3cb48fb1 >2014-08-25 15:16:10,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-28) [b86371b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3cb48fb1 >2014-08-25 15:16:10,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [b86371b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-25 15:16:10,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [b86371b] Correlation ID: 1c912a72, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-25 15:16:10,657 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [6fc3f838] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:44:48,032 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-25 15:44:48,033 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-25 15:44:48,034 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-25 15:44:48,035 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-25 15:44:48,036 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-25 15:44:48,037 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-25 15:44:48,038 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-25 15:44:48,039 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-25 15:44:48,040 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-25 15:44:48,064 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-25 15:44:48,065 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-25 15:44:48,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-25 15:44:48,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-25 15:44:48,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-25 15:44:48,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-25 15:44:48,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-25 15:44:48,069 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-25 15:44:48,070 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-25 15:44:48,070 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-25 15:44:48,071 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-25 15:44:48,072 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-25 15:44:48,072 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-25 15:44:48,073 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-25 15:44:48,074 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-25 15:44:48,113 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-25 15:44:48,113 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-25 15:44:48,114 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-25 15:44:48,115 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-25 15:44:48,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-25 15:44:48,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-25 15:44:48,117 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-25 15:44:48,117 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-25 15:44:48,118 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-25 15:44:48,119 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-25 15:44:48,119 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-25 15:44:48,120 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-25 15:44:48,121 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-25 15:44:48,121 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-25 15:44:48,123 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-25 15:44:48,123 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-25 15:44:48,149 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-25 15:44:48,149 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-25 15:44:48,150 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-25 15:44:48,151 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-25 15:44:48,151 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-25 15:44:48,152 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-25 15:44:48,153 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-25 15:44:48,153 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-25 15:44:48,154 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-25 15:44:48,155 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-25 15:44:48,155 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-25 15:44:48,156 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-25 15:44:48,157 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-25 15:44:48,158 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-25 15:44:48,186 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-25 15:44:48,187 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-25 15:44:48,188 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-25 15:44:48,188 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-25 15:44:48,189 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-25 15:44:48,190 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-25 15:44:48,190 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-25 15:44:48,191 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-25 15:44:48,192 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-25 15:44:48,192 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-25 15:44:48,193 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-25 15:44:48,194 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-25 15:44:48,194 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-25 15:44:48,195 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-25 15:44:51,973 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-25 15:44:53,837 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-25 15:44:53,838 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-25 15:44:54,215 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140821064931.gitb794d66.el6 >2014-08-25 15:44:54,216 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-25 15:44:54,221 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-25 15:44:54,222 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-25 15:44:54,222 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-25 15:44:54,337 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-25 15:44:54,337 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-25 15:44:54,338 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-25 15:44:54,341 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-25 15:44:54,343 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-25 15:44:54,406 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-25 15:44:54,442 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-25 15:44:54,443 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-25 15:44:54,452 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-25 15:44:54,747 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-25 15:44:54,890 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-25 15:44:54,961 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-25 15:44:54,963 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-25 15:44:54,964 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-25 15:44:55,006 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-25 15:44:55,007 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-25 15:44:55,067 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-25 15:44:55,180 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-25 15:44:55,182 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-25 15:44:55,182 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-25 15:44:55,184 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-25 15:44:55,187 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-25 15:44:55,196 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-25 15:44:55,362 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-25 15:44:55,363 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 15:44:55,365 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 15:44:55,366 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-25 15:44:55,760 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-1) Start initializing PmHealthCheckManager >2014-08-25 15:44:55,761 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-1) Finished initializing PmHealthCheckManager >2014-08-25 15:44:55,763 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-25 15:44:55,764 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-25 15:44:56,094 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-25 15:44:56,097 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-25 15:44:56,445 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-25 15:44:56,619 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 15:44:56,704 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-25 15:44:57,058 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-25 15:44:57,059 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 15:44:57,101 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-25 15:44:57,124 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-25 15:44:57,125 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 15:44:57,154 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-25 15:44:57,180 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-25 15:44:57,243 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-25 15:44:57,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-25 15:44:57,263 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-25 15:44:57,264 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-25 15:44:57,265 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-25 15:44:57,266 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-25 15:44:57,314 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-25 15:44:57,315 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-25 15:44:57,321 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-25 15:44:57,328 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-25 15:44:57,353 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-25 15:44:57,427 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-25 15:44:57,428 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-25 15:44:57,429 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-25 15:44:57,431 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-25 15:45:02,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 774d878a >2014-08-25 15:45:02,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 1958c103 >2014-08-25 15:45:02,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 478cc0c0 >2014-08-25 15:45:02,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 1958c103 >2014-08-25 15:45:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, GetHardwareInfoVDSCommand, log id: 774d878a >2014-08-25 15:45:02,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 478cc0c0 >2014-08-25 15:45:03,729 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-25 15:45:03,756 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-25 15:45:03,757 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:03,792 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-25 15:45:03,840 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: UNASSIGNED not exist in string table >2014-08-25 15:45:03,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-25 15:45:03,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-25 15:45:03,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-25 15:45:03,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-25 15:45:03,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-25 15:45:03,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-25 15:45:03,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-25 15:45:03,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-25 15:45:03,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-25 15:45:03,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STOPPED not exist in string table >2014-08-25 15:45:03,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STARTED not exist in string table >2014-08-25 15:45:03,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_ERROR not exist in string table >2014-08-25 15:45:03,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-25 15:45:03,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-25 15:45:03,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-25 15:45:03,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-25 15:45:03,886 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [696ae80c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-25 15:45:03,881 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-25 15:45:04,036 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [5be86992] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-25 15:45:04,107 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [46a0276d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 15:45:04,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 15:45:07,702 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-25) [25d2341] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 15:45:07,714 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [25d2341] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-25 15:45:08,073 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [25d2341] Correlation ID: 25d2341, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-25 15:45:08,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [25d2341] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-25 15:45:08,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [25d2341] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-25 15:45:08,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [25d2341] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-25 15:45:08,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [25d2341] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 400a7f81 >2014-08-25 15:45:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [25d2341] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-25 15:45:08,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [25d2341] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 2869d046 >2014-08-25 15:45:09,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [25d2341] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2869d046 >2014-08-25 15:45:09,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [25d2341] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 400a7f81 >2014-08-25 15:45:09,019 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) [25d2341] Discovered no tasks on Storage Pool DC1 >2014-08-25 15:45:15,762 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-47) Power Management Health Check started. >2014-08-25 15:45:16,144 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginAdminUserCommand internal: false. >2014-08-25 15:45:16,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 15:45:16,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:16,292 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:16,389 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:16,412 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-25 15:45:16,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 111a50f9 >2014-08-25 15:45:17,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 111a50f9 >2014-08-25 15:45:17,354 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-47) Power Management Health Check completed. >2014-08-25 15:45:37,356 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 15:45:37,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:37,714 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:37,773 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:37,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 34892c52 >2014-08-25 15:45:38,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 34892c52 >2014-08-25 15:45:38,131 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 15:45:49,160 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:45:49,165 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:45:49,167 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [5277beae] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-08-25 15:45:49,341 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:45:49,342 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:45:49,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:49,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:49,948 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:49,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 3282ad91 >2014-08-25 15:45:51,022 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:45:51,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3282ad91 >2014-08-25 15:45:51,024 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:45:51,512 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:45:51,513 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:45:51,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:51,722 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:51,723 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:51,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 5bf32a80 >2014-08-25 15:45:52,060 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:45:52,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5bf32a80 >2014-08-25 15:45:52,108 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:45:52,162 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 5277beae, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-25 15:45:58,135 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-94) Power Management Health Check started. >2014-08-25 15:45:58,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:58,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:58,512 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:58,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 45349ab1 >2014-08-25 15:45:58,826 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:45:58,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 45349ab1 >2014-08-25 15:45:58,828 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:45:59,322 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:45:59,323 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:45:59,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:45:59,413 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:45:59,414 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-94) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:45:59,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 735d41fb >2014-08-25 15:45:59,701 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:45:59,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 735d41fb >2014-08-25 15:45:59,760 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:45:59,761 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-94) Power Management Health Check completed. >2014-08-25 15:46:19,763 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check started. >2014-08-25 15:46:20,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:20,110 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:20,143 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:46:20,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: f25017d >2014-08-25 15:46:20,422 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:20,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: f25017d >2014-08-25 15:46:20,424 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:46:20,588 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:46:20,588 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:46:20,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:20,659 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:20,660 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:46:20,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 7b045751 >2014-08-25 15:46:20,900 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:20,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7b045751 >2014-08-25 15:46:20,976 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:46:20,977 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check completed. >2014-08-25 15:46:40,977 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check started. >2014-08-25 15:46:41,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:41,123 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:41,165 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:46:41,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 1a182a4b >2014-08-25 15:46:41,391 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:41,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a182a4b >2014-08-25 15:46:41,393 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:46:41,546 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:46:41,547 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:46:41,619 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:41,620 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:41,621 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-m325mt.qa.lab.tlv.redhat.com, User:root, Options:power_wait=20, Fencing policy:null >2014-08-25 15:46:41,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-m325mt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'power_wait=20', policy = 'null'), log id: 263f8361 >2014-08-25 15:46:41,864 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:41,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-m325mt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 263f8361 >2014-08-25 15:46:41,906 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:46:41,907 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check completed. >2014-08-25 15:46:57,525 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:46:57,539 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:46:57,540 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [70c0323a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-08-25 15:46:57,573 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:46:57,574 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:46:57,907 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:57,908 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:57,940 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:46:57,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: db20ff2 >2014-08-25 15:46:58,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:58,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: db20ff2 >2014-08-25 15:46:58,179 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:46:58,357 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:46:58,358 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:46:58,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:46:58,457 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:46:58,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:46:58,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 313bfce >2014-08-25 15:46:58,710 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:58,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 313bfce >2014-08-25 15:46:58,756 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:46:58,761 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: 70c0323a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-25 15:47:01,908 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-1) Power Management Health Check started. >2014-08-25 15:47:02,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:02,177 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:02,226 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:02,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 67ac845b >2014-08-25 15:47:02,542 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:02,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 67ac845b >2014-08-25 15:47:02,554 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:47:02,993 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:47:02,994 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:47:03,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:03,141 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:03,142 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:03,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4af7c6b3 >2014-08-25 15:47:03,446 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:03,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4af7c6b3 >2014-08-25 15:47:03,526 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:47:03,527 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-1) Power Management Health Check completed. >2014-08-25 15:47:23,528 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check started. >2014-08-25 15:47:23,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:23,718 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:23,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:23,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 30895fc1 >2014-08-25 15:47:24,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:24,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 30895fc1 >2014-08-25 15:47:24,045 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:47:24,212 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:47:24,212 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:47:24,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:24,342 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:24,343 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:24,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6836e8ed >2014-08-25 15:47:24,595 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:24,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6836e8ed >2014-08-25 15:47:24,635 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:47:24,636 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check completed. >2014-08-25 15:47:44,637 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-41) Power Management Health Check started. >2014-08-25 15:47:44,818 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:44,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:44,851 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:44,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 267e00b1 >2014-08-25 15:47:45,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:45,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 267e00b1 >2014-08-25 15:47:45,080 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:47:45,252 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:47:45,252 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:47:45,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:47:45,330 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:47:45,331 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:47:45,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6e0579d9 >2014-08-25 15:47:45,617 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:47:45,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6e0579d9 >2014-08-25 15:47:45,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:47:45,659 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-41) Power Management Health Check completed. >2014-08-25 15:48:05,660 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 15:48:05,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:05,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:05,878 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:05,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a5438d9 >2014-08-25 15:48:06,313 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:06,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a5438d9 >2014-08-25 15:48:06,316 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:48:06,452 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:48:06,453 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:48:06,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:06,530 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:06,531 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:06,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5191ce5b >2014-08-25 15:48:06,789 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:06,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5191ce5b >2014-08-25 15:48:06,838 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:48:06,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 15:48:26,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check started. >2014-08-25 15:48:27,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:27,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:27,143 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:27,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7741267e >2014-08-25 15:48:27,408 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:27,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7741267e >2014-08-25 15:48:27,411 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:48:27,557 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:48:27,558 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:48:27,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:27,634 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:27,635 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:27,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 44dcad7f >2014-08-25 15:48:27,876 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:27,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 44dcad7f >2014-08-25 15:48:27,973 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:48:27,974 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check completed. >2014-08-25 15:48:30,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:30,210 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 15:48:30,263 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:30,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5b781377 >2014-08-25 15:48:30,567 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:30,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5b781377 >2014-08-25 15:48:30,569 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:48:30,745 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to run Power Management command on Host , no running proxy Host was found. >2014-08-25 15:48:30,745 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:48:30,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:30,817 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 15:48:30,818 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:30,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 659328c0 >2014-08-25 15:48:31,070 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:31,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 659328c0 >2014-08-25 15:48:39,013 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:48:39,014 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:48:39,015 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [39ad4432] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-08-25 15:48:39,086 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:48:39,087 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-25 15:48:39,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:39,383 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:39,414 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:39,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 356c2792 >2014-08-25 15:48:39,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:39,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 356c2792 >2014-08-25 15:48:39,660 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:48:39,841 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:48:39,842 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:48:39,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:39,926 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:39,927 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:39,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 73395136 >2014-08-25 15:48:40,163 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:40,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 73395136 >2014-08-25 15:48:40,201 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:40,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 39ad4432, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-25 15:48:47,976 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check started. >2014-08-25 15:48:48,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:48,109 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:48,140 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:48,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2aa3ed4c >2014-08-25 15:48:48,375 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:48,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2aa3ed4c >2014-08-25 15:48:48,377 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:48:48,507 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:48:48,508 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:48:48,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:48:48,576 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:48:48,577 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:48:48,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 37e7c327 >2014-08-25 15:48:48,811 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:48:48,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 37e7c327 >2014-08-25 15:48:48,861 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:48:48,862 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check completed. >2014-08-25 15:49:08,863 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check started. >2014-08-25 15:49:09,016 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:09,018 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:09,050 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:09,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c484801 >2014-08-25 15:49:09,440 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:09,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c484801 >2014-08-25 15:49:09,443 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:49:09,672 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:49:09,673 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:49:09,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:09,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:09,844 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:09,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 453c7cfb >2014-08-25 15:49:10,121 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:10,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 453c7cfb >2014-08-25 15:49:10,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:49:10,193 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check completed. >2014-08-25 15:49:30,194 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check started. >2014-08-25 15:49:30,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:30,344 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:30,390 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:30,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1bb2d001 >2014-08-25 15:49:30,625 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:30,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1bb2d001 >2014-08-25 15:49:30,627 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:49:30,768 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:49:30,769 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:49:30,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:30,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:30,926 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:30,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f8362f5 >2014-08-25 15:49:31,215 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:31,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f8362f5 >2014-08-25 15:49:31,263 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:49:31,264 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check completed. >2014-08-25 15:49:51,265 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check started. >2014-08-25 15:49:51,394 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:51,396 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:51,427 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:51,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e2808b1 >2014-08-25 15:49:51,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:51,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e2808b1 >2014-08-25 15:49:51,695 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:49:51,859 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:49:51,859 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:49:51,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:49:51,957 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:49:51,958 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:49:51,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 78d0caa0 >2014-08-25 15:49:52,225 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:49:52,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 78d0caa0 >2014-08-25 15:49:52,262 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:49:52,263 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check completed. >2014-08-25 15:50:12,264 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check started. >2014-08-25 15:50:12,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:12,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:12,425 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:12,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5023d94d >2014-08-25 15:50:12,707 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:12,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5023d94d >2014-08-25 15:50:12,709 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:50:12,848 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:50:12,849 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:50:12,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:12,916 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:12,917 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:12,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 66f0cf7 >2014-08-25 15:50:13,165 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:13,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 66f0cf7 >2014-08-25 15:50:13,201 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:50:13,202 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check completed. >2014-08-25 15:50:33,204 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 15:50:33,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:33,339 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:33,372 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:33,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 596e55bc >2014-08-25 15:50:34,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:34,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 596e55bc >2014-08-25 15:50:34,195 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:50:34,325 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:50:34,326 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:50:34,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:34,451 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:34,452 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:34,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 58161ad8 >2014-08-25 15:50:34,699 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:34,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 58161ad8 >2014-08-25 15:50:34,777 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:50:34,779 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 15:50:54,782 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check started. >2014-08-25 15:50:55,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:55,090 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:55,133 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:55,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2d197994 >2014-08-25 15:50:55,442 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:55,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2d197994 >2014-08-25 15:50:55,444 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:50:55,695 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:50:55,695 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:50:55,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:50:55,838 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:50:55,839 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:50:55,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2d33de89 >2014-08-25 15:50:56,130 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:50:56,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2d33de89 >2014-08-25 15:50:56,199 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:50:56,201 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check completed. >2014-08-25 15:51:16,201 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-52) Power Management Health Check started. >2014-08-25 15:51:16,363 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:16,364 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:16,397 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:16,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 729202d5 >2014-08-25 15:51:16,650 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:16,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 729202d5 >2014-08-25 15:51:16,652 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:51:16,782 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:51:16,783 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:51:16,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:16,924 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:16,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:16,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2dc4f315 >2014-08-25 15:51:17,295 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:17,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2dc4f315 >2014-08-25 15:51:17,342 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:51:17,344 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-52) Power Management Health Check completed. >2014-08-25 15:51:37,345 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 15:51:37,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:37,484 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:37,531 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:37,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: fb97571 >2014-08-25 15:51:37,948 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:37,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: fb97571 >2014-08-25 15:51:37,951 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:51:38,081 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:51:38,082 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:51:38,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:38,151 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:38,152 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:38,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 522b7408 >2014-08-25 15:51:38,486 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:38,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 522b7408 >2014-08-25 15:51:38,521 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:51:38,523 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 15:51:58,523 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-4) Power Management Health Check started. >2014-08-25 15:51:58,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:58,652 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:58,684 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:58,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 30c035ab >2014-08-25 15:51:58,926 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:58,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 30c035ab >2014-08-25 15:51:58,928 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:51:59,060 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:51:59,061 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:51:59,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:51:59,146 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:51:59,147 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:51:59,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 80de803 >2014-08-25 15:51:59,407 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:51:59,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 80de803 >2014-08-25 15:51:59,443 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:51:59,444 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-4) Power Management Health Check completed. >2014-08-25 15:52:19,446 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check started. >2014-08-25 15:52:19,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:52:19,581 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:52:19,616 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:52:19,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 37ad8512 >2014-08-25 15:52:19,842 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:52:19,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 37ad8512 >2014-08-25 15:52:19,845 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:52:19,975 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:52:19,976 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:52:20,099 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:52:20,100 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:52:20,101 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:52:20,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 117d52f2 >2014-08-25 15:52:20,330 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:52:20,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 117d52f2 >2014-08-25 15:52:20,381 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:52:20,382 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check completed. >2014-08-25 15:52:40,384 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check started. >2014-08-25 15:52:40,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:52:40,529 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:52:40,560 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:52:40,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 73553e31 >2014-08-25 15:52:40,786 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:52:40,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 73553e31 >2014-08-25 15:52:40,789 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:52:40,923 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:52:40,924 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:52:40,991 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:52:40,992 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:52:40,993 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:52:40,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 15683b8c >2014-08-25 15:52:41,236 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:52:41,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 15683b8c >2014-08-25 15:52:41,287 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:52:41,288 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check completed. >2014-08-25 15:53:01,288 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check started. >2014-08-25 15:53:01,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:01,419 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:01,450 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:01,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 63449e7d >2014-08-25 15:53:01,779 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:01,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 63449e7d >2014-08-25 15:53:01,781 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:53:01,922 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:53:01,922 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:53:01,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:01,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:01,991 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:01,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 22b90dc6 >2014-08-25 15:53:02,310 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:02,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 22b90dc6 >2014-08-25 15:53:02,346 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:53:02,347 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check completed. >2014-08-25 15:53:22,348 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check started. >2014-08-25 15:53:22,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:22,477 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:22,522 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:22,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ba80808 >2014-08-25 15:53:22,755 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:22,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ba80808 >2014-08-25 15:53:22,758 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:53:22,915 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:53:22,915 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:53:23,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:23,029 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:23,030 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:23,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 719518ed >2014-08-25 15:53:23,262 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:23,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 719518ed >2014-08-25 15:53:23,298 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:53:23,299 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check completed. >2014-08-25 15:53:43,308 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-48) Power Management Health Check started. >2014-08-25 15:53:43,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:43,759 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:43,823 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:43,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7cd8b034 >2014-08-25 15:53:44,198 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:44,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7cd8b034 >2014-08-25 15:53:44,201 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:53:44,510 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:53:44,511 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:53:44,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:53:44,677 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:53:44,678 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:53:44,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 47b8f52d >2014-08-25 15:53:44,957 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:53:44,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 47b8f52d >2014-08-25 15:53:45,028 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:53:45,029 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-48) Power Management Health Check completed. >2014-08-25 15:54:05,031 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 15:54:05,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:05,193 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:05,225 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:05,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4afa1c93 >2014-08-25 15:54:05,474 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:05,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4afa1c93 >2014-08-25 15:54:05,477 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:54:05,606 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:54:05,607 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:54:05,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:05,673 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:05,674 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:05,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5175ca9a >2014-08-25 15:54:05,919 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:05,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5175ca9a >2014-08-25 15:54:05,963 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:54:05,964 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 15:54:25,966 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check started. >2014-08-25 15:54:26,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:26,096 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:26,159 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:26,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6ba17eaa >2014-08-25 15:54:26,467 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:26,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6ba17eaa >2014-08-25 15:54:26,469 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:54:26,609 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:54:26,609 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:54:26,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:26,676 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:26,676 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:26,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ff6542a >2014-08-25 15:54:26,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:26,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ff6542a >2014-08-25 15:54:26,964 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:54:26,965 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check completed. >2014-08-25 15:54:46,966 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-63) Power Management Health Check started. >2014-08-25 15:54:47,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:47,096 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:47,127 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:47,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 52521323 >2014-08-25 15:54:47,354 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:47,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 52521323 >2014-08-25 15:54:47,356 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:54:47,486 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:54:47,486 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:54:47,566 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:54:47,567 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:54:47,568 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:54:47,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: e44ac77 >2014-08-25 15:54:47,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:54:47,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: e44ac77 >2014-08-25 15:54:47,866 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:54:47,868 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-63) Power Management Health Check completed. >2014-08-25 15:55:07,868 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check started. >2014-08-25 15:55:08,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:08,005 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:08,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:08,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7e1ad85a >2014-08-25 15:55:08,269 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:08,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7e1ad85a >2014-08-25 15:55:08,272 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:55:08,433 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:55:08,434 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:55:08,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:08,510 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:08,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:08,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: e31fa91 >2014-08-25 15:55:08,771 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:08,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: e31fa91 >2014-08-25 15:55:08,817 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:55:08,819 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check completed. >2014-08-25 15:55:28,820 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) [696ae80c] Power Management Health Check started. >2014-08-25 15:55:28,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [696ae80c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:28,951 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:28,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:28,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [696ae80c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 381b5340 >2014-08-25 15:55:29,208 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [696ae80c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:29,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [696ae80c] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 381b5340 >2014-08-25 15:55:29,210 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:55:29,351 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:55:29,351 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:55:29,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [696ae80c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:29,418 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:29,420 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) [696ae80c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:29,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [696ae80c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: c4ae425 >2014-08-25 15:55:29,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [696ae80c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:29,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [696ae80c] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: c4ae425 >2014-08-25 15:55:29,706 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [696ae80c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:55:29,707 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) [696ae80c] Power Management Health Check completed. >2014-08-25 15:55:49,717 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check started. >2014-08-25 15:55:49,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:49,951 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:49,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:49,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3d34b332 >2014-08-25 15:55:50,761 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:50,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3d34b332 >2014-08-25 15:55:50,763 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:55:50,891 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:55:50,892 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:55:50,957 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:55:50,958 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:55:50,959 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:55:50,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 40b8cf82 >2014-08-25 15:55:51,195 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:55:51,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 40b8cf82 >2014-08-25 15:55:51,230 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:55:51,232 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check completed. >2014-08-25 15:56:11,239 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check started. >2014-08-25 15:56:11,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:11,411 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:11,475 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:11,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4da5b6c6 >2014-08-25 15:56:11,764 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:11,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4da5b6c6 >2014-08-25 15:56:11,766 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:56:12,016 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:56:12,017 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:56:12,169 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:12,170 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:12,171 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:12,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ca00c5 >2014-08-25 15:56:12,458 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:12,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ca00c5 >2014-08-25 15:56:12,503 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:56:12,504 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check completed. >2014-08-25 15:56:32,505 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 15:56:32,632 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:32,633 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:32,702 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:32,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 620e454 >2014-08-25 15:56:32,944 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:32,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 620e454 >2014-08-25 15:56:32,946 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:56:33,091 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:56:33,092 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:56:33,250 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:33,251 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:33,252 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:33,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 65cb0225 >2014-08-25 15:56:33,490 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:33,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 65cb0225 >2014-08-25 15:56:33,531 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:56:33,532 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 15:56:53,534 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check started. >2014-08-25 15:56:53,664 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:53,665 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:53,696 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:53,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 616389f7 >2014-08-25 15:56:53,935 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:53,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 616389f7 >2014-08-25 15:56:53,937 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:56:54,079 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:56:54,080 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:56:54,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:56:54,198 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:56:54,199 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:56:54,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 73487628 >2014-08-25 15:56:54,425 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:56:54,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 73487628 >2014-08-25 15:56:54,460 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:56:54,461 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check completed. >2014-08-25 15:57:14,463 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check started. >2014-08-25 15:57:14,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:14,611 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:14,643 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:14,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f0b9d32 >2014-08-25 15:57:14,876 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:14,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f0b9d32 >2014-08-25 15:57:14,878 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:57:15,007 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:57:15,008 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:57:15,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:15,076 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:15,076 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:15,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 76323be5 >2014-08-25 15:57:15,308 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:15,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 76323be5 >2014-08-25 15:57:15,351 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:57:15,352 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check completed. >2014-08-25 15:57:35,353 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check started. >2014-08-25 15:57:35,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:35,500 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:35,531 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:35,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1f13735b >2014-08-25 15:57:35,774 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:35,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1f13735b >2014-08-25 15:57:35,777 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:57:35,941 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:57:35,942 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:57:36,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:36,053 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:36,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:36,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 15901f91 >2014-08-25 15:57:36,285 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:36,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 15901f91 >2014-08-25 15:57:36,342 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:57:36,344 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check completed. >2014-08-25 15:57:56,345 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check started. >2014-08-25 15:57:56,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:56,477 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:56,508 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:56,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: a8760eb >2014-08-25 15:57:56,736 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:56,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: a8760eb >2014-08-25 15:57:56,739 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:57:56,871 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:57:56,872 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:57:56,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:57:56,942 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:57:56,943 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:57:56,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 204a80f7 >2014-08-25 15:57:57,191 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:57:57,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 204a80f7 >2014-08-25 15:57:57,228 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:57:57,229 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check completed. >2014-08-25 15:58:17,230 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check started. >2014-08-25 15:58:17,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:17,359 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:17,391 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:17,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 622c789e >2014-08-25 15:58:17,636 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:17,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 622c789e >2014-08-25 15:58:17,638 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:58:17,770 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:58:17,770 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:58:17,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:17,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:17,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:17,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: a4925da >2014-08-25 15:58:18,074 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:18,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: a4925da >2014-08-25 15:58:18,110 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:58:18,111 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check completed. >2014-08-25 15:58:38,112 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check started. >2014-08-25 15:58:38,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:38,246 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:38,277 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:38,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7ea9091f >2014-08-25 15:58:38,532 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:38,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7ea9091f >2014-08-25 15:58:38,534 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:58:38,693 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:58:38,694 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:58:38,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:38,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:38,790 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:38,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50c30ecf >2014-08-25 15:58:39,018 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:39,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50c30ecf >2014-08-25 15:58:39,054 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:58:39,056 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check completed. >2014-08-25 15:58:59,056 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 15:58:59,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:59,189 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:59,221 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:59,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7a9a2731 >2014-08-25 15:58:59,527 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:59,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7a9a2731 >2014-08-25 15:58:59,530 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:58:59,659 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:58:59,660 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:58:59,726 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:58:59,727 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:58:59,728 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:58:59,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 668d8594 >2014-08-25 15:58:59,982 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:58:59,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 668d8594 >2014-08-25 15:59:00,017 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:59:00,018 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 15:59:20,019 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check started. >2014-08-25 15:59:20,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:59:20,149 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:59:20,180 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:59:20,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1b6c5c72 >2014-08-25 15:59:20,415 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:59:20,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1b6c5c72 >2014-08-25 15:59:20,419 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:59:20,552 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:59:20,553 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:59:20,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:59:20,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:59:20,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:59:20,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 682da620 >2014-08-25 15:59:20,845 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:59:20,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 682da620 >2014-08-25 15:59:20,881 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:59:20,883 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check completed. >2014-08-25 15:59:40,883 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check started. >2014-08-25 15:59:41,010 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:59:41,011 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:59:41,042 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:59:41,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1c2782bc >2014-08-25 15:59:41,275 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:59:41,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1c2782bc >2014-08-25 15:59:41,277 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 15:59:41,406 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 15:59:41,407 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 15:59:41,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 15:59:41,474 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 15:59:41,475 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 15:59:41,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 122e7dbb >2014-08-25 15:59:41,698 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 15:59:41,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 122e7dbb >2014-08-25 15:59:41,734 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 15:59:41,735 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check completed. >2014-08-25 16:00:01,736 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check started. >2014-08-25 16:00:01,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:00:01,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:00:01,962 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:00:01,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 69530dd2 >2014-08-25 16:00:02,224 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:00:02,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 69530dd2 >2014-08-25 16:00:02,226 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:00:02,354 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:00:02,355 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:00:02,421 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:00:02,422 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:00:02,423 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:00:02,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 297fcff3 >2014-08-25 16:00:02,650 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:00:02,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 297fcff3 >2014-08-25 16:00:02,686 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:00:02,688 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check completed. >2014-08-25 16:00:22,688 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 16:00:22,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:00:22,823 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:00:22,853 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:00:22,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6d6fb290 >2014-08-25 16:00:23,106 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:00:23,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6d6fb290 >2014-08-25 16:00:23,108 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:00:23,245 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:00:23,246 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:00:23,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:00:23,313 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:00:23,314 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:00:23,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7c80511a >2014-08-25 16:00:23,598 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:00:23,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7c80511a >2014-08-25 16:00:23,667 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:00:23,668 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 16:00:55,327 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-25 16:00:55,328 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-25 16:00:55,329 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-25 16:00:55,330 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-25 16:00:55,331 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-25 16:00:55,332 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-25 16:00:55,333 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-25 16:00:55,334 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-25 16:00:55,335 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-25 16:00:55,336 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-25 16:00:55,336 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-25 16:00:55,337 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-25 16:00:55,370 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-25 16:00:55,371 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-25 16:00:55,371 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-25 16:00:55,372 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-25 16:00:55,373 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-25 16:00:55,373 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-25 16:00:55,374 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-25 16:00:55,375 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-25 16:00:55,375 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-25 16:00:55,376 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-25 16:00:55,377 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-25 16:00:55,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-25 16:00:55,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-25 16:00:55,379 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-25 16:00:55,379 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-25 16:00:55,402 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-25 16:00:55,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-25 16:00:55,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-25 16:00:55,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-25 16:00:55,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-25 16:00:55,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-25 16:00:55,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-25 16:00:55,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-25 16:00:55,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-25 16:00:55,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-25 16:00:55,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-25 16:00:55,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-25 16:00:55,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-25 16:00:55,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-25 16:00:55,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-25 16:00:55,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-25 16:00:55,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-25 16:00:55,426 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-25 16:00:55,427 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-25 16:00:55,427 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-25 16:00:55,428 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-25 16:00:55,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-25 16:00:55,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-25 16:00:55,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-25 16:00:55,431 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-25 16:00:55,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-25 16:00:55,433 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-25 16:00:55,433 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-25 16:00:55,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-25 16:00:55,435 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-25 16:00:55,435 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-25 16:00:55,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-25 16:00:55,449 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-25 16:00:55,450 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-25 16:00:55,451 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-25 16:00:55,452 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-25 16:00:55,452 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-25 16:00:55,453 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-25 16:00:55,453 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-25 16:00:55,454 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-25 16:00:55,455 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-25 16:00:57,099 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-25 16:00:58,249 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-25 16:00:58,251 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-25 16:00:58,528 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140821064931.gitb794d66.el6 >2014-08-25 16:00:58,529 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-25 16:00:58,534 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-25 16:00:58,534 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-25 16:00:58,535 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-25 16:00:58,611 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-25 16:00:58,612 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-25 16:00:58,613 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-25 16:00:58,615 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-25 16:00:58,618 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-25 16:00:58,637 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-25 16:00:58,665 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-25 16:00:58,666 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-25 16:00:58,687 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-25 16:00:58,936 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-25 16:00:59,011 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-25 16:00:59,050 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-25 16:00:59,052 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-25 16:00:59,053 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-25 16:00:59,070 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-25 16:00:59,071 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-25 16:00:59,149 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-25 16:00:59,151 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-25 16:00:59,151 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-25 16:00:59,154 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-25 16:00:59,156 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-25 16:00:59,184 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-25 16:00:59,315 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-25 16:00:59,316 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 16:00:59,318 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 16:00:59,319 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-25 16:00:59,626 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-2) Start initializing PmHealthCheckManager >2014-08-25 16:00:59,627 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-2) Finished initializing PmHealthCheckManager >2014-08-25 16:00:59,628 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-25 16:00:59,629 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-25 16:00:59,669 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-25 16:00:59,672 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-25 16:00:59,964 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 16:01:00,045 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-25 16:01:00,388 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-25 16:01:00,512 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-25 16:01:00,512 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 16:01:00,540 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-25 16:01:00,567 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-25 16:01:00,567 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-25 16:01:00,606 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-25 16:01:00,636 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-25 16:01:00,700 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-25 16:01:00,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-25 16:01:00,707 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-25 16:01:00,722 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-25 16:01:00,722 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-25 16:01:00,723 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-25 16:01:00,746 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-25 16:01:00,747 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-25 16:01:00,764 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-25 16:01:00,787 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-25 16:01:00,807 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-25 16:01:00,885 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-25 16:01:00,886 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-25 16:01:00,886 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-25 16:01:00,888 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-25 16:01:05,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 620dc788 >2014-08-25 16:01:06,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 59fa6956 >2014-08-25 16:01:06,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 620dc788 >2014-08-25 16:01:06,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 59fa6956 >2014-08-25 16:01:06,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 34534588 >2014-08-25 16:01:06,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 34534588 >2014-08-25 16:01:06,920 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-25 16:01:06,911 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-25 16:01:07,199 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-25 16:01:07,417 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 16:01:07,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-25 16:01:07,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-25 16:01:10,921 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 16:01:10,964 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [534aca73] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-25 16:01:11,008 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-25 16:01:11,018 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: UNASSIGNED not exist in string table >2014-08-25 16:01:11,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-25 16:01:11,020 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-25 16:01:11,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-25 16:01:11,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-25 16:01:11,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-25 16:01:11,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-25 16:01:11,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-25 16:01:11,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-25 16:01:11,027 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-25 16:01:11,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: DWH_STOPPED not exist in string table >2014-08-25 16:01:11,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: DWH_STARTED not exist in string table >2014-08-25 16:01:11,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: DWH_ERROR not exist in string table >2014-08-25 16:01:11,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-25 16:01:11,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-25 16:01:11,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-25 16:01:11,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-25 16:01:11,614 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: 534aca73, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-25 16:01:11,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [534aca73] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-25 16:01:12,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [534aca73] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-25 16:01:12,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-25 16:01:12,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [534aca73] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 1ce5f5a4 >2014-08-25 16:01:12,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [534aca73] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-25 16:01:12,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [534aca73] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 9007850 >2014-08-25 16:01:12,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [534aca73] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9007850 >2014-08-25 16:01:12,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [534aca73] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ce5f5a4 >2014-08-25 16:01:12,303 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [534aca73] Discovered no tasks on Storage Pool DC1 >2014-08-25 16:01:19,627 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-46) Power Management Health Check started. >2014-08-25 16:01:20,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:01:20,166 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:01:20,246 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:01:20,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4296c9fd >2014-08-25 16:01:21,170 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:01:21,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4296c9fd >2014-08-25 16:01:21,173 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:01:21,439 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:01:21,440 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:01:21,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:01:21,570 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:01:21,571 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:01:21,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e5d0ea3 >2014-08-25 16:01:21,851 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:01:21,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e5d0ea3 >2014-08-25 16:01:21,908 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:01:21,909 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-46) Power Management Health Check completed. >2014-08-25 16:01:41,911 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check started. >2014-08-25 16:01:42,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:01:42,294 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:01:42,349 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:01:42,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a512208 >2014-08-25 16:01:42,757 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:01:42,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a512208 >2014-08-25 16:01:42,760 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:01:43,099 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:01:43,100 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:01:43,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:01:43,219 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:01:43,220 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:01:43,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 78d583fd >2014-08-25 16:01:43,499 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:01:43,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 78d583fd >2014-08-25 16:01:43,550 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:01:43,551 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check completed. >2014-08-25 16:02:03,552 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check started. >2014-08-25 16:02:03,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:03,854 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:03,908 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:03,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51ad67e7 >2014-08-25 16:02:04,351 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:04,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51ad67e7 >2014-08-25 16:02:04,353 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:02:04,515 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:02:04,528 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:02:04,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:04,622 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:04,623 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:04,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 47101710 >2014-08-25 16:02:04,861 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:04,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 47101710 >2014-08-25 16:02:04,907 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:02:04,908 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check completed. >2014-08-25 16:02:24,910 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 16:02:25,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:25,143 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:25,179 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:25,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7ea4003 >2014-08-25 16:02:25,417 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:25,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7ea4003 >2014-08-25 16:02:25,420 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:02:25,611 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:02:25,622 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:02:25,764 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:25,765 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:25,766 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:25,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: c14a82a >2014-08-25 16:02:26,000 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:26,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: c14a82a >2014-08-25 16:02:26,058 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:02:26,059 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 16:02:46,061 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check started. >2014-08-25 16:02:46,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:46,260 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:46,293 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:46,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: f5dca1a >2014-08-25 16:02:46,529 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:46,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: f5dca1a >2014-08-25 16:02:46,531 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:02:46,692 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:02:46,693 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:02:46,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:02:46,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:02:46,778 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:02:46,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5b8e5893 >2014-08-25 16:02:47,032 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:02:47,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5b8e5893 >2014-08-25 16:02:47,081 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:02:47,082 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check completed. >2014-08-25 16:03:07,084 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check started. >2014-08-25 16:03:07,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:07,221 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:07,254 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:07,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5a5ae84b >2014-08-25 16:03:07,493 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:07,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5a5ae84b >2014-08-25 16:03:07,495 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:03:07,649 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:03:07,650 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:03:07,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:07,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:07,725 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:07,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51254f93 >2014-08-25 16:03:08,088 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:08,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51254f93 >2014-08-25 16:03:08,125 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:03:08,126 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check completed. >2014-08-25 16:03:28,128 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check started. >2014-08-25 16:03:28,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:28,293 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:28,325 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:28,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 33d93a5a >2014-08-25 16:03:28,652 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:28,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 33d93a5a >2014-08-25 16:03:28,654 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:03:28,801 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:03:28,802 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:03:28,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:28,941 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:28,942 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:28,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f85b2e0 >2014-08-25 16:03:29,189 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:29,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f85b2e0 >2014-08-25 16:03:29,234 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:03:29,235 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check completed. >2014-08-25 16:03:49,237 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check started. >2014-08-25 16:03:49,391 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:49,392 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:49,424 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:49,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: a40564c >2014-08-25 16:03:49,703 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:49,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: a40564c >2014-08-25 16:03:49,705 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:03:49,878 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:03:49,879 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:03:50,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:03:50,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:03:50,004 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:03:50,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1ae9f669 >2014-08-25 16:03:50,284 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:03:50,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1ae9f669 >2014-08-25 16:03:50,326 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:03:50,327 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check completed. >2014-08-25 16:04:10,329 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check started. >2014-08-25 16:04:10,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:10,525 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:10,557 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:10,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 29760b0f >2014-08-25 16:04:10,815 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:10,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 29760b0f >2014-08-25 16:04:10,817 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:04:11,001 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:04:11,002 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:04:11,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:11,135 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:11,137 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:11,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: d041f8c >2014-08-25 16:04:11,384 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:11,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: d041f8c >2014-08-25 16:04:11,421 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:04:11,422 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check completed. >2014-08-25 16:04:31,424 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check started. >2014-08-25 16:04:31,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:31,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:31,650 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:31,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 46f105c >2014-08-25 16:04:32,081 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:32,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 46f105c >2014-08-25 16:04:32,084 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:04:32,270 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:04:32,273 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:04:32,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:32,341 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:32,342 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:32,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 14d25e6c >2014-08-25 16:04:32,584 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:32,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 14d25e6c >2014-08-25 16:04:32,633 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:04:32,634 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check completed. >2014-08-25 16:04:52,635 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check started. >2014-08-25 16:04:52,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:52,793 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:52,838 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:52,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7c23861c >2014-08-25 16:04:53,073 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:53,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7c23861c >2014-08-25 16:04:53,076 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:04:53,273 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:04:53,274 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:04:53,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:04:53,342 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:04:53,343 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:04:53,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 9a2c811 >2014-08-25 16:04:53,640 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:04:53,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 9a2c811 >2014-08-25 16:04:53,712 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:04:53,713 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check completed. >2014-08-25 16:05:13,715 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check started. >2014-08-25 16:05:13,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:13,922 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:13,965 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:13,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 376f5c1a >2014-08-25 16:05:14,224 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:14,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 376f5c1a >2014-08-25 16:05:14,226 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:05:14,377 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:05:14,378 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:05:14,603 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:14,604 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:14,605 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:14,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f4a2efa >2014-08-25 16:05:15,005 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:15,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f4a2efa >2014-08-25 16:05:15,079 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:05:15,080 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check completed. >2014-08-25 16:05:35,082 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-2) Power Management Health Check started. >2014-08-25 16:05:35,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:35,284 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:35,317 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:35,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2dbb99c2 >2014-08-25 16:05:35,571 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:35,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2dbb99c2 >2014-08-25 16:05:35,573 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:05:35,755 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:05:35,756 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:05:35,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:35,825 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:35,826 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-2) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:35,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1bb7aeaa >2014-08-25 16:05:36,059 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:36,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1bb7aeaa >2014-08-25 16:05:36,107 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:05:36,109 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-2) Power Management Health Check completed. >2014-08-25 16:05:56,110 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check started. >2014-08-25 16:05:56,262 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:56,264 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:56,311 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:56,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 23268a92 >2014-08-25 16:05:56,539 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:56,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 23268a92 >2014-08-25 16:05:56,541 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:05:56,675 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:05:56,676 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:05:56,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:05:56,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:05:56,745 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:05:56,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 116e2eb5 >2014-08-25 16:05:56,976 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:05:56,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 116e2eb5 >2014-08-25 16:05:57,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:05:57,043 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check completed. >2014-08-25 16:06:17,045 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check started. >2014-08-25 16:06:17,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:06:17,177 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:06:17,209 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:06:17,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 244ad77e >2014-08-25 16:06:17,487 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:06:17,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 244ad77e >2014-08-25 16:06:17,490 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:06:17,669 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:06:17,670 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:06:17,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:06:17,749 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:06:17,750 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:06:17,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2d6688a >2014-08-25 16:06:18,010 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:06:18,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2d6688a >2014-08-25 16:06:18,048 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:06:18,049 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check completed. >2014-08-25 16:06:38,052 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check started. >2014-08-25 16:06:38,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:06:38,222 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:06:38,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:06:38,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2dde8c0e >2014-08-25 16:06:38,984 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:06:38,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2dde8c0e >2014-08-25 16:06:38,987 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:06:39,149 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:06:39,150 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:06:39,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:06:39,230 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:06:39,231 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:06:39,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 24b01e43 >2014-08-25 16:06:39,470 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:06:39,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 24b01e43 >2014-08-25 16:06:39,507 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:06:39,509 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check completed. >2014-08-25 16:06:59,509 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check started. >2014-08-25 16:06:59,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:06:59,680 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:06:59,719 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:06:59,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5ab6e003 >2014-08-25 16:07:00,036 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:00,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5ab6e003 >2014-08-25 16:07:00,038 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:07:00,178 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:07:00,179 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:07:00,261 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:07:00,262 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:07:00,263 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:07:00,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 23764d2d >2014-08-25 16:07:00,496 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:00,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 23764d2d >2014-08-25 16:07:00,533 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:07:00,534 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check completed. >2014-08-25 16:07:20,536 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-58) Power Management Health Check started. >2014-08-25 16:07:20,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:07:20,674 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:07:20,706 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:07:20,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6b32a76f >2014-08-25 16:07:21,120 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:21,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6b32a76f >2014-08-25 16:07:21,122 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:07:21,274 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:07:21,275 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:07:21,345 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:07:21,346 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:07:21,347 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-58) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:07:21,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 54e38971 >2014-08-25 16:07:21,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:21,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 54e38971 >2014-08-25 16:07:21,695 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:07:21,696 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-58) Power Management Health Check completed. >2014-08-25 16:07:41,697 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check started. >2014-08-25 16:07:41,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:07:41,833 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:07:41,867 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:07:41,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1844e791 >2014-08-25 16:07:42,161 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:42,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1844e791 >2014-08-25 16:07:42,163 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:07:42,292 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:07:42,293 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:07:42,360 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:07:42,361 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:07:42,362 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:07:42,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 229b05c5 >2014-08-25 16:07:42,641 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:07:42,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 229b05c5 >2014-08-25 16:07:42,679 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:07:42,680 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check completed. >2014-08-25 16:08:02,682 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-6) Power Management Health Check started. >2014-08-25 16:08:02,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:02,814 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:02,845 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:02,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1d685ac4 >2014-08-25 16:08:03,164 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:03,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1d685ac4 >2014-08-25 16:08:03,167 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:08:03,297 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:08:03,298 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:08:03,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:03,380 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:03,381 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:03,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57884b0c >2014-08-25 16:08:03,648 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:03,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57884b0c >2014-08-25 16:08:03,727 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:08:03,729 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-6) Power Management Health Check completed. >2014-08-25 16:08:23,729 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 16:08:23,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:23,866 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:23,900 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:23,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 58d6338 >2014-08-25 16:08:24,129 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:24,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 58d6338 >2014-08-25 16:08:24,131 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:08:24,273 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:08:24,273 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:08:24,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:24,342 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:24,343 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:24,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 792178df >2014-08-25 16:08:24,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:24,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 792178df >2014-08-25 16:08:24,701 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:08:24,703 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 16:08:44,705 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check started. >2014-08-25 16:08:44,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:44,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:45,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:45,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3590f66f >2014-08-25 16:08:45,316 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:45,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3590f66f >2014-08-25 16:08:45,318 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:08:45,543 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:08:45,544 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:08:45,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:08:45,618 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:08:45,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:08:45,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5170702f >2014-08-25 16:08:45,861 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:08:45,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5170702f >2014-08-25 16:08:45,904 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:08:45,906 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check completed. >2014-08-25 16:09:05,907 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check started. >2014-08-25 16:09:06,068 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:06,070 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:06,102 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:06,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: c16e5d >2014-08-25 16:09:06,334 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:06,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: c16e5d >2014-08-25 16:09:06,337 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:09:06,481 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:09:06,482 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:09:06,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:06,580 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:06,580 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:06,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c41fe20 >2014-08-25 16:09:06,808 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:06,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c41fe20 >2014-08-25 16:09:06,849 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:09:06,850 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check completed. >2014-08-25 16:09:26,851 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check started. >2014-08-25 16:09:27,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:27,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:27,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:27,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 77a83226 >2014-08-25 16:09:27,262 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:27,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 77a83226 >2014-08-25 16:09:27,264 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:09:27,392 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:09:27,393 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:09:27,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:27,479 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:27,480 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:27,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c29498f >2014-08-25 16:09:27,710 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:27,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c29498f >2014-08-25 16:09:27,751 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:09:27,752 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check completed. >2014-08-25 16:09:47,752 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check started. >2014-08-25 16:09:47,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:47,883 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:47,914 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:47,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3cb463 >2014-08-25 16:09:48,337 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:48,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3cb463 >2014-08-25 16:09:48,339 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:09:48,482 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:09:48,483 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:09:48,549 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:09:48,550 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:09:48,551 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:09:48,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 72fbf782 >2014-08-25 16:09:48,792 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:09:48,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 72fbf782 >2014-08-25 16:09:48,828 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:09:48,829 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check completed. >2014-08-25 16:10:08,830 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check started. >2014-08-25 16:10:09,000 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:09,001 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:09,042 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:09,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1f1a9e38 >2014-08-25 16:10:09,281 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:09,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1f1a9e38 >2014-08-25 16:10:09,284 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:10:09,419 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:10:09,420 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:10:09,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:09,497 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:09,498 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:09,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51383a10 >2014-08-25 16:10:09,821 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:09,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51383a10 >2014-08-25 16:10:09,857 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:10:09,858 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check completed. >2014-08-25 16:10:29,859 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-97) Power Management Health Check started. >2014-08-25 16:10:29,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:29,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:30,021 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:30,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 40d60e9 >2014-08-25 16:10:30,260 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:30,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 40d60e9 >2014-08-25 16:10:30,262 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:10:30,401 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:10:30,402 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:10:30,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:30,476 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:30,477 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:30,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 748b2492 >2014-08-25 16:10:30,711 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:30,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 748b2492 >2014-08-25 16:10:30,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:10:30,799 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-97) Power Management Health Check completed. >2014-08-25 16:10:50,799 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-64) Power Management Health Check started. >2014-08-25 16:10:50,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:50,934 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:50,964 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:50,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 93db9e3 >2014-08-25 16:10:51,202 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:51,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 93db9e3 >2014-08-25 16:10:51,204 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:10:51,349 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:10:51,349 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:10:51,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:10:51,417 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:10:51,418 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:10:51,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 67ddb166 >2014-08-25 16:10:51,646 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:10:51,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 67ddb166 >2014-08-25 16:10:51,708 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:10:51,709 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-64) Power Management Health Check completed. >2014-08-25 16:11:11,710 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check started. >2014-08-25 16:11:11,840 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:11,842 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:11,873 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:11,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 370b91b5 >2014-08-25 16:11:12,100 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:12,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 370b91b5 >2014-08-25 16:11:12,103 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:11:12,231 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:11:12,232 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:11:12,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:12,301 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:12,302 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:12,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 93d03ed >2014-08-25 16:11:12,558 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:12,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 93d03ed >2014-08-25 16:11:12,594 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:11:12,596 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check completed. >2014-08-25 16:11:32,596 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check started. >2014-08-25 16:11:32,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:32,728 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:32,760 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:32,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 14c6b8db >2014-08-25 16:11:33,121 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:33,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 14c6b8db >2014-08-25 16:11:33,123 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:11:33,253 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:11:33,254 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:11:33,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:33,323 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:33,324 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:33,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6495b3c >2014-08-25 16:11:33,561 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:33,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6495b3c >2014-08-25 16:11:33,597 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:11:33,599 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check completed. >2014-08-25 16:11:53,600 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-77) Power Management Health Check started. >2014-08-25 16:11:53,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:53,747 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:53,780 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:53,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4c5c6b36 >2014-08-25 16:11:54,744 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:54,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4c5c6b36 >2014-08-25 16:11:54,747 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:11:54,877 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:11:54,877 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:11:54,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:11:54,944 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:11:54,944 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:11:54,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4014b71c >2014-08-25 16:11:55,200 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:11:55,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4014b71c >2014-08-25 16:11:55,236 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:11:55,238 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-77) Power Management Health Check completed. >2014-08-25 16:12:15,239 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:12:15,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:15,372 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:15,403 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:15,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 27ba380f >2014-08-25 16:12:15,630 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:15,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 27ba380f >2014-08-25 16:12:15,633 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:12:15,765 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:12:15,766 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:12:15,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:15,856 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:15,857 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:15,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 661f6b42 >2014-08-25 16:12:16,117 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:16,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 661f6b42 >2014-08-25 16:12:16,154 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:12:16,156 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:12:36,157 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 16:12:36,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:36,306 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:36,337 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:36,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6554419a >2014-08-25 16:12:36,631 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:36,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6554419a >2014-08-25 16:12:36,634 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:12:36,821 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:12:36,821 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:12:36,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:36,889 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:36,890 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:36,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 75a65e0f >2014-08-25 16:12:37,123 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:37,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 75a65e0f >2014-08-25 16:12:37,159 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:12:37,161 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 16:12:57,163 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check started. >2014-08-25 16:12:57,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:57,323 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:57,353 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:57,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6db74a2a >2014-08-25 16:12:57,587 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:57,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6db74a2a >2014-08-25 16:12:57,589 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:12:57,721 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:12:57,722 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:12:57,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:12:57,790 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:12:57,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:12:57,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 285c948d >2014-08-25 16:12:58,023 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:12:58,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 285c948d >2014-08-25 16:12:58,059 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:12:58,060 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check completed. >2014-08-25 16:13:18,061 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check started. >2014-08-25 16:13:18,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:13:18,192 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:13:18,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:13:18,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a94738c >2014-08-25 16:13:18,519 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:13:18,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a94738c >2014-08-25 16:13:18,522 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:13:18,662 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:13:18,663 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:13:18,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:13:18,760 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:13:18,760 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:13:18,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57406da3 >2014-08-25 16:13:19,082 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:13:19,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57406da3 >2014-08-25 16:13:19,153 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:13:19,155 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check completed. >2014-08-25 16:13:39,155 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 16:13:39,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:13:39,292 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:13:39,323 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:13:39,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 195da3b7 >2014-08-25 16:13:39,561 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:13:39,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 195da3b7 >2014-08-25 16:13:39,563 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:13:39,699 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:13:39,699 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:13:39,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:13:39,772 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:13:39,773 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:13:39,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 45a6e4b0 >2014-08-25 16:13:40,099 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:13:40,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 45a6e4b0 >2014-08-25 16:13:40,136 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:13:40,138 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 16:14:00,139 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-3) Power Management Health Check started. >2014-08-25 16:14:00,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:00,275 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:00,307 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:00,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 751eacf >2014-08-25 16:14:00,536 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:00,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 751eacf >2014-08-25 16:14:00,538 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:14:00,674 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:14:00,675 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:14:00,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:00,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:00,844 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:00,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 116f9901 >2014-08-25 16:14:01,201 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:01,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 116f9901 >2014-08-25 16:14:01,273 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:14:01,274 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-3) Power Management Health Check completed. >2014-08-25 16:14:21,276 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check started. >2014-08-25 16:14:21,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:21,429 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:21,465 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:21,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 25d6c21 >2014-08-25 16:14:21,700 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:21,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 25d6c21 >2014-08-25 16:14:21,703 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:14:21,835 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:14:21,836 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:14:21,922 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:21,924 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:21,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:21,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5902178a >2014-08-25 16:14:22,154 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:22,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5902178a >2014-08-25 16:14:22,195 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:14:22,197 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check completed. >2014-08-25 16:14:42,198 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-47) Power Management Health Check started. >2014-08-25 16:14:42,325 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:42,326 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:42,363 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:42,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 164c9a1a >2014-08-25 16:14:42,597 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:42,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 164c9a1a >2014-08-25 16:14:42,600 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:14:42,740 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:14:42,740 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:14:42,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:14:42,809 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:14:42,810 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-47) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:14:42,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 426332a0 >2014-08-25 16:14:43,041 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:14:43,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 426332a0 >2014-08-25 16:14:43,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:14:43,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-47) Power Management Health Check completed. >2014-08-25 16:15:03,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check started. >2014-08-25 16:15:03,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:03,206 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:03,267 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:03,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e553b90 >2014-08-25 16:15:03,501 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:03,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e553b90 >2014-08-25 16:15:03,503 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:15:03,735 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:15:03,736 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:15:03,804 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:03,805 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:03,806 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:03,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3189741c >2014-08-25 16:15:04,038 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:04,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3189741c >2014-08-25 16:15:04,075 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:15:04,077 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check completed. >2014-08-25 16:15:24,077 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-89) Power Management Health Check started. >2014-08-25 16:15:24,215 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:24,217 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:24,249 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:24,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 755b23a9 >2014-08-25 16:15:24,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:24,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 755b23a9 >2014-08-25 16:15:24,478 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:15:24,607 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:15:24,608 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:15:24,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:24,712 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:24,713 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:24,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3dc27960 >2014-08-25 16:15:24,943 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:24,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3dc27960 >2014-08-25 16:15:25,076 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:15:25,077 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-89) Power Management Health Check completed. >2014-08-25 16:15:45,078 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:15:45,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:45,208 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:45,239 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:45,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 43082ca6 >2014-08-25 16:15:45,472 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:45,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 43082ca6 >2014-08-25 16:15:45,475 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:15:45,604 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:15:45,605 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:15:45,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:15:45,672 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:15:45,673 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:15:45,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 74e82135 >2014-08-25 16:15:45,901 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:15:45,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 74e82135 >2014-08-25 16:15:45,939 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:15:45,940 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:16:05,941 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check started. >2014-08-25 16:16:06,193 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:06,194 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:06,248 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:06,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5103e2ce >2014-08-25 16:16:06,516 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:06,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5103e2ce >2014-08-25 16:16:06,518 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:16:06,650 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:16:06,651 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:16:06,724 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:06,725 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:06,726 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:06,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 42938297 >2014-08-25 16:16:06,955 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:06,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 42938297 >2014-08-25 16:16:06,991 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:16:06,992 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check completed. >2014-08-25 16:16:26,993 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-100) Power Management Health Check started. >2014-08-25 16:16:27,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:27,123 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:27,156 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:27,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51d3810 >2014-08-25 16:16:27,480 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:27,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51d3810 >2014-08-25 16:16:27,483 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:16:27,614 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:16:27,615 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:16:27,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:27,682 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:27,683 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:27,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79c31765 >2014-08-25 16:16:27,916 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:27,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79c31765 >2014-08-25 16:16:27,952 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:16:27,953 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-100) Power Management Health Check completed. >2014-08-25 16:16:47,954 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check started. >2014-08-25 16:16:48,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:48,094 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:48,125 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:48,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50b5a28c >2014-08-25 16:16:48,357 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:48,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50b5a28c >2014-08-25 16:16:48,361 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:16:48,496 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:16:48,497 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:16:48,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:16:48,566 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:16:48,567 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:16:48,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51a42281 >2014-08-25 16:16:48,801 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:16:48,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51a42281 >2014-08-25 16:16:48,837 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:16:48,838 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check completed. >2014-08-25 16:17:08,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check started. >2014-08-25 16:17:08,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:08,973 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:09,005 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:09,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f8dc4fb >2014-08-25 16:17:09,928 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:09,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f8dc4fb >2014-08-25 16:17:09,930 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:17:10,096 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:17:10,097 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:17:10,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:10,166 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:10,166 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:10,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1e3f9540 >2014-08-25 16:17:10,398 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:10,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1e3f9540 >2014-08-25 16:17:10,434 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:17:10,435 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check completed. >2014-08-25 16:17:30,435 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-3) Power Management Health Check started. >2014-08-25 16:17:30,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:30,567 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:30,598 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:30,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a61b683 >2014-08-25 16:17:30,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:30,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a61b683 >2014-08-25 16:17:30,833 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:17:30,967 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:17:30,967 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:17:31,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:31,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:31,037 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:31,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 24181281 >2014-08-25 16:17:31,422 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:31,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 24181281 >2014-08-25 16:17:31,460 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:17:31,461 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-3) Power Management Health Check completed. >2014-08-25 16:17:51,463 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check started. >2014-08-25 16:17:51,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:51,596 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:51,631 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:51,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 113877f8 >2014-08-25 16:17:51,869 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:51,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 113877f8 >2014-08-25 16:17:51,871 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:17:51,998 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:17:51,999 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:17:52,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:17:52,066 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:17:52,067 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:17:52,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 400042cb >2014-08-25 16:17:52,299 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:17:52,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 400042cb >2014-08-25 16:17:52,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:17:52,336 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check completed. >2014-08-25 16:18:12,337 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:18:12,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:12,472 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:12,505 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:12,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 75d56d0a >2014-08-25 16:18:12,743 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:12,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 75d56d0a >2014-08-25 16:18:12,746 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:18:12,981 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:18:12,982 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:18:13,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:13,053 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:13,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:13,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: bcae4b7 >2014-08-25 16:18:13,284 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:13,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: bcae4b7 >2014-08-25 16:18:13,319 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:18:13,320 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:18:33,321 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check started. >2014-08-25 16:18:33,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:33,493 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:33,525 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:33,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1da42653 >2014-08-25 16:18:33,755 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:33,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1da42653 >2014-08-25 16:18:33,758 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:18:33,924 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:18:33,925 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:18:33,995 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:33,996 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:33,997 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:33,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6f4be588 >2014-08-25 16:18:34,229 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:34,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6f4be588 >2014-08-25 16:18:34,264 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:18:34,266 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check completed. >2014-08-25 16:18:54,266 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check started. >2014-08-25 16:18:54,394 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:54,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:54,426 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:54,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 564f6f3f >2014-08-25 16:18:54,653 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:54,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 564f6f3f >2014-08-25 16:18:54,656 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:18:54,826 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:18:54,826 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:18:54,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:18:54,897 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:18:54,898 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:18:54,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 32cf4c72 >2014-08-25 16:18:55,138 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:18:55,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 32cf4c72 >2014-08-25 16:18:55,215 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:18:55,216 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check completed. >2014-08-25 16:19:15,217 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check started. >2014-08-25 16:19:15,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:15,352 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:15,382 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:15,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 12e1d0a9 >2014-08-25 16:19:15,611 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:15,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 12e1d0a9 >2014-08-25 16:19:15,614 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:19:15,741 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:19:15,742 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:19:15,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:15,810 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:15,811 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:15,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 16b4d4eb >2014-08-25 16:19:16,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:16,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 16b4d4eb >2014-08-25 16:19:16,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:19:16,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check completed. >2014-08-25 16:19:36,080 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check started. >2014-08-25 16:19:36,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:36,226 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:36,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:36,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 98c3720 >2014-08-25 16:19:36,543 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:36,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 98c3720 >2014-08-25 16:19:36,546 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:19:36,677 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:19:36,678 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:19:36,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:36,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:36,745 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:36,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c624362 >2014-08-25 16:19:37,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:37,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c624362 >2014-08-25 16:19:37,069 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:19:37,070 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check completed. >2014-08-25 16:19:57,071 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-1) Power Management Health Check started. >2014-08-25 16:19:57,199 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:57,200 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:57,232 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:57,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 35c98900 >2014-08-25 16:19:57,456 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:57,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 35c98900 >2014-08-25 16:19:57,458 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:19:57,599 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:19:57,599 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:19:57,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:19:57,667 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:19:57,668 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:19:57,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7e48414e >2014-08-25 16:19:57,904 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:19:57,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7e48414e >2014-08-25 16:19:57,979 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:19:57,980 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-1) Power Management Health Check completed. >2014-08-25 16:20:17,981 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-64) Power Management Health Check started. >2014-08-25 16:20:18,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:20:18,147 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:20:18,190 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:20:18,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 58a385b6 >2014-08-25 16:20:18,415 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:20:18,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 58a385b6 >2014-08-25 16:20:18,417 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:20:18,545 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:20:18,545 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:20:18,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:20:18,611 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:20:18,612 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-64) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:20:18,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 403a0144 >2014-08-25 16:20:18,840 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:20:18,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 403a0144 >2014-08-25 16:20:18,875 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:20:18,877 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-64) Power Management Health Check completed. >2014-08-25 16:20:38,878 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-42) Power Management Health Check started. >2014-08-25 16:20:39,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:20:39,061 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:20:39,103 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:20:39,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 341fb46f >2014-08-25 16:20:39,343 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:20:39,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 341fb46f >2014-08-25 16:20:39,346 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:20:39,473 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:20:39,473 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:20:39,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:20:39,540 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:20:39,541 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:20:39,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 277c23a5 >2014-08-25 16:20:39,768 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:20:39,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 277c23a5 >2014-08-25 16:20:39,804 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:20:39,805 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-42) Power Management Health Check completed. >2014-08-25 16:20:59,806 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check started. >2014-08-25 16:20:59,935 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:20:59,937 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:20:59,968 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:20:59,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ad3d6b6 >2014-08-25 16:21:00,197 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ad3d6b6 >2014-08-25 16:21:00,200 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:21:00,345 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:21:00,346 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:21:00,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:21:00,413 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:21:00,414 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:21:00,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 31e5b434 >2014-08-25 16:21:00,640 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 31e5b434 >2014-08-25 16:21:00,676 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:21:00,677 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check completed. >2014-08-25 16:21:20,679 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-86) Power Management Health Check started. >2014-08-25 16:21:20,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:21:20,832 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:21:20,863 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:21:20,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 61449375 >2014-08-25 16:21:21,094 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:21,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 61449375 >2014-08-25 16:21:21,097 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:21:21,228 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:21:21,229 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:21:21,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:21:21,306 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:21:21,307 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:21:21,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2dd174bd >2014-08-25 16:21:21,783 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:21,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2dd174bd >2014-08-25 16:21:21,819 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:21:21,820 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-86) Power Management Health Check completed. >2014-08-25 16:21:41,822 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check started. >2014-08-25 16:21:41,974 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:21:41,976 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:21:42,007 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:21:42,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7a5d642e >2014-08-25 16:21:42,231 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:42,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7a5d642e >2014-08-25 16:21:42,234 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:21:42,362 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:21:42,363 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:21:42,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:21:42,429 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:21:42,430 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:21:42,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79e7a314 >2014-08-25 16:21:42,781 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:21:42,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79e7a314 >2014-08-25 16:21:42,817 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:21:42,819 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check completed. >2014-08-25 16:22:02,820 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check started. >2014-08-25 16:22:02,957 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:02,958 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:02,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:02,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6724a1ab >2014-08-25 16:22:03,392 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:03,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6724a1ab >2014-08-25 16:22:03,395 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:22:03,523 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:22:03,523 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:22:03,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:03,594 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:03,595 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:03,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6e9ff829 >2014-08-25 16:22:03,820 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:03,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6e9ff829 >2014-08-25 16:22:03,863 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:22:03,864 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check completed. >2014-08-25 16:22:23,864 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check started. >2014-08-25 16:22:23,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:23,995 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:24,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:24,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7c3a9f69 >2014-08-25 16:22:24,792 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:24,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7c3a9f69 >2014-08-25 16:22:24,794 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:22:24,921 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:22:24,922 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:22:24,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:24,988 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:24,989 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:24,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 406b15ad >2014-08-25 16:22:25,229 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:25,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 406b15ad >2014-08-25 16:22:25,265 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:22:25,267 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check completed. >2014-08-25 16:22:45,268 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-74) Power Management Health Check started. >2014-08-25 16:22:45,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:45,397 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:45,428 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:45,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4d6d24f7 >2014-08-25 16:22:45,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:45,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4d6d24f7 >2014-08-25 16:22:45,660 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:22:45,854 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:22:45,855 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:22:45,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:22:45,929 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:22:45,930 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:22:45,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: e44570 >2014-08-25 16:22:46,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:22:46,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: e44570 >2014-08-25 16:22:46,199 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:22:46,201 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-74) Power Management Health Check completed. >2014-08-25 16:23:06,201 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check started. >2014-08-25 16:23:06,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:06,330 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:06,392 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:06,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 9d45adc >2014-08-25 16:23:06,741 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:06,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 9d45adc >2014-08-25 16:23:06,743 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:23:06,907 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:23:06,908 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:23:06,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:06,981 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:06,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:06,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 492406e8 >2014-08-25 16:23:07,210 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:07,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 492406e8 >2014-08-25 16:23:07,246 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:23:07,247 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check completed. >2014-08-25 16:23:27,247 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check started. >2014-08-25 16:23:27,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:27,380 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:27,412 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:27,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1e41505d >2014-08-25 16:23:27,639 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:27,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1e41505d >2014-08-25 16:23:27,642 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:23:27,768 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:23:27,769 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:23:27,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:27,866 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:27,867 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:27,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 63f64f4d >2014-08-25 16:23:28,091 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:28,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 63f64f4d >2014-08-25 16:23:28,126 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:23:28,128 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check completed. >2014-08-25 16:23:48,129 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check started. >2014-08-25 16:23:48,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:48,257 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:48,318 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:48,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5d07f78b >2014-08-25 16:23:48,543 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:48,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5d07f78b >2014-08-25 16:23:48,545 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:23:48,670 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:23:48,671 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:23:48,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:23:48,739 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:23:48,740 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:23:48,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 274d5cba >2014-08-25 16:23:48,964 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:23:48,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 274d5cba >2014-08-25 16:23:48,999 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:23:49,000 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check completed. >2014-08-25 16:24:09,002 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check started. >2014-08-25 16:24:09,130 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:09,131 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:09,162 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:09,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 70cd7f42 >2014-08-25 16:24:09,556 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:09,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 70cd7f42 >2014-08-25 16:24:09,559 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:24:09,689 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:24:09,690 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:24:09,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:09,757 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:09,758 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:09,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e5b9234 >2014-08-25 16:24:09,989 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:09,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e5b9234 >2014-08-25 16:24:10,025 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:24:10,026 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check completed. >2014-08-25 16:24:30,028 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-25) [534aca73] Power Management Health Check started. >2014-08-25 16:24:30,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:30,157 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:30,189 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:30,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79680a2e >2014-08-25 16:24:30,422 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:30,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79680a2e >2014-08-25 16:24:30,425 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:24:30,555 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:24:30,556 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:24:30,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:30,624 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:30,625 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:30,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 38727af0 >2014-08-25 16:24:30,860 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:30,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 38727af0 >2014-08-25 16:24:30,896 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:24:30,898 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-25) [534aca73] Power Management Health Check completed. >2014-08-25 16:24:50,899 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check started. >2014-08-25 16:24:51,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:51,028 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:51,058 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:51,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 427d7783 >2014-08-25 16:24:51,282 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:51,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 427d7783 >2014-08-25 16:24:51,285 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:24:51,418 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:24:51,419 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:24:51,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:24:51,486 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:24:51,487 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:24:51,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e3886a >2014-08-25 16:24:51,712 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:24:51,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e3886a >2014-08-25 16:24:51,747 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:24:51,749 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check completed. >2014-08-25 16:25:11,749 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check started. >2014-08-25 16:25:11,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:11,877 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:11,909 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:11,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2674fd36 >2014-08-25 16:25:12,135 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:12,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2674fd36 >2014-08-25 16:25:12,138 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:25:12,270 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:25:12,271 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:25:12,336 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:12,337 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:12,338 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:12,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6229e01b >2014-08-25 16:25:12,563 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:12,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6229e01b >2014-08-25 16:25:12,601 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:25:12,602 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check completed. >2014-08-25 16:25:32,603 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check started. >2014-08-25 16:25:32,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:32,730 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:32,762 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:32,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 36718a67 >2014-08-25 16:25:33,015 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:33,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 36718a67 >2014-08-25 16:25:33,018 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:25:33,173 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:25:33,174 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:25:33,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:33,247 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:33,248 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-34) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:33,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6ab786ef >2014-08-25 16:25:33,474 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:33,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6ab786ef >2014-08-25 16:25:33,520 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:25:33,522 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-34) Power Management Health Check completed. >2014-08-25 16:25:53,524 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 16:25:53,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:53,651 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:53,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:53,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 751fb515 >2014-08-25 16:25:53,950 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:53,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 751fb515 >2014-08-25 16:25:53,953 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:25:54,086 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:25:54,086 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:25:54,151 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:25:54,153 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:25:54,153 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:25:54,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 461ffd54 >2014-08-25 16:25:54,382 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:25:54,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 461ffd54 >2014-08-25 16:25:54,418 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:25:54,419 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 16:26:14,420 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check started. >2014-08-25 16:26:14,618 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:14,620 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:14,651 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:14,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f8221eb >2014-08-25 16:26:14,879 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:14,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f8221eb >2014-08-25 16:26:14,881 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:26:15,011 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:26:15,012 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:26:15,077 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:15,078 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:15,079 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:15,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 23dfa5fb >2014-08-25 16:26:15,307 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:15,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 23dfa5fb >2014-08-25 16:26:15,363 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:26:15,365 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check completed. >2014-08-25 16:26:35,365 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check started. >2014-08-25 16:26:35,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:35,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:35,543 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:35,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 728027a9 >2014-08-25 16:26:35,809 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:35,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 728027a9 >2014-08-25 16:26:35,812 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:26:35,945 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:26:35,945 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:26:36,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:36,012 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:36,013 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:36,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1212f00c >2014-08-25 16:26:36,242 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:36,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1212f00c >2014-08-25 16:26:36,277 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:26:36,278 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check completed. >2014-08-25 16:26:56,279 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check started. >2014-08-25 16:26:56,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:56,408 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:56,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:56,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3134328f >2014-08-25 16:26:56,667 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:56,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3134328f >2014-08-25 16:26:56,669 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:26:56,795 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:26:56,796 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:26:56,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:26:56,864 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:26:56,865 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:26:56,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c00d4cd >2014-08-25 16:26:57,392 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:26:57,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c00d4cd >2014-08-25 16:26:57,427 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:26:57,429 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-11) Power Management Health Check completed. >2014-08-25 16:27:17,429 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check started. >2014-08-25 16:27:17,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:27:17,558 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:27:17,588 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:27:17,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 158cf0ef >2014-08-25 16:27:17,908 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:27:17,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 158cf0ef >2014-08-25 16:27:17,910 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:27:18,037 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:27:18,037 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:27:18,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:27:18,112 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:27:18,113 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:27:18,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ea14f9a >2014-08-25 16:27:18,342 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:27:18,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ea14f9a >2014-08-25 16:27:18,379 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:27:18,380 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check completed. >2014-08-25 16:27:38,382 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check started. >2014-08-25 16:27:38,547 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:27:38,548 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:27:38,579 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:27:38,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7dc6b466 >2014-08-25 16:27:39,356 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:27:39,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7dc6b466 >2014-08-25 16:27:39,359 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:27:39,488 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:27:39,489 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:27:39,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:27:39,556 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:27:39,557 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:27:39,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 22789345 >2014-08-25 16:27:39,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:27:39,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 22789345 >2014-08-25 16:27:39,832 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:27:39,833 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check completed. >2014-08-25 16:27:59,834 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check started. >2014-08-25 16:28:00,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:00,049 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:00,080 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:00,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4ff1ab12 >2014-08-25 16:28:00,365 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:00,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4ff1ab12 >2014-08-25 16:28:00,367 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:28:00,534 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:28:00,535 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:28:00,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:00,605 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:00,606 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 988301b >2014-08-25 16:28:00,836 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:00,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 988301b >2014-08-25 16:28:00,876 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:28:00,877 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check completed. >2014-08-25 16:28:20,801 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginAdminUserCommand internal: false. >2014-08-25 16:28:20,818 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 16:28:20,885 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check started. >2014-08-25 16:28:21,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:21,413 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:21,524 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:21,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7ccef785 >2014-08-25 16:28:21,944 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:21,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7ccef785 >2014-08-25 16:28:21,947 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:28:22,467 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:28:22,468 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:28:22,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:22,835 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:22,836 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:22,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2e4a5501 >2014-08-25 16:28:23,211 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:23,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2e4a5501 >2014-08-25 16:28:23,369 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:28:23,370 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check completed. >2014-08-25 16:28:30,190 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-25 16:28:30,214 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 16:28:43,371 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check started. >2014-08-25 16:28:43,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:43,527 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:43,557 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:43,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3d35dd6a >2014-08-25 16:28:43,825 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:43,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3d35dd6a >2014-08-25 16:28:43,828 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:28:44,182 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:28:44,183 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:28:44,352 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:28:44,353 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:28:44,354 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:28:44,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: fce8450 >2014-08-25 16:28:44,632 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:28:44,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: fce8450 >2014-08-25 16:28:44,713 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:28:44,714 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check completed. >2014-08-25 16:29:04,720 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check started. >2014-08-25 16:29:04,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:04,920 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:04,951 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:04,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 74b0afc8 >2014-08-25 16:29:05,175 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:05,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 74b0afc8 >2014-08-25 16:29:05,177 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:29:05,360 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:29:05,361 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:29:05,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:05,543 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:05,544 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:05,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 672fecd8 >2014-08-25 16:29:05,784 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:05,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 672fecd8 >2014-08-25 16:29:05,820 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:29:05,821 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check completed. >2014-08-25 16:29:25,822 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-21) Power Management Health Check started. >2014-08-25 16:29:25,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:25,949 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:25,981 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:25,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 142b3f1d >2014-08-25 16:29:26,261 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:26,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 142b3f1d >2014-08-25 16:29:26,264 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:29:26,404 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:29:26,405 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:29:26,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:26,471 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:26,472 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-21) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:26,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 43f07021 >2014-08-25 16:29:26,699 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:26,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 43f07021 >2014-08-25 16:29:26,734 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:29:26,736 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-21) Power Management Health Check completed. >2014-08-25 16:29:46,736 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check started. >2014-08-25 16:29:46,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:46,883 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:46,917 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:46,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4e99ec17 >2014-08-25 16:29:47,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:47,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4e99ec17 >2014-08-25 16:29:47,165 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:29:47,291 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:29:47,292 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:29:47,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:29:47,359 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:29:47,360 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:29:47,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 8f8c502 >2014-08-25 16:29:47,747 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:29:47,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 8f8c502 >2014-08-25 16:29:47,783 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:29:47,784 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check completed. >2014-08-25 16:30:07,786 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check started. >2014-08-25 16:30:07,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:07,916 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:07,947 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:07,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3845a815 >2014-08-25 16:30:08,174 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:08,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3845a815 >2014-08-25 16:30:08,177 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:30:08,306 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:30:08,306 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:30:08,372 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:08,373 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:08,374 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:08,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5c2d5b06 >2014-08-25 16:30:08,600 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:08,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5c2d5b06 >2014-08-25 16:30:08,647 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:30:08,649 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check completed. >2014-08-25 16:30:28,650 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-30) Power Management Health Check started. >2014-08-25 16:30:28,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:28,779 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:28,810 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:28,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 248b4d61 >2014-08-25 16:30:29,063 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:29,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 248b4d61 >2014-08-25 16:30:29,065 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:30:29,197 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:30:29,198 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:30:29,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:29,265 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:29,266 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:29,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3bf26c17 >2014-08-25 16:30:29,605 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:29,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3bf26c17 >2014-08-25 16:30:29,641 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:30:29,642 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-30) Power Management Health Check completed. >2014-08-25 16:30:49,642 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 16:30:49,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:49,795 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:49,827 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:49,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3ea70fce >2014-08-25 16:30:50,059 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:50,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3ea70fce >2014-08-25 16:30:50,062 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:30:50,189 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:30:50,190 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:30:50,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:30:50,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:30:50,290 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:30:50,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 59ecb30f >2014-08-25 16:30:50,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:30:50,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 59ecb30f >2014-08-25 16:30:50,707 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:30:50,708 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 16:31:10,710 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 16:31:10,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:10,853 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:10,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:10,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1e151475 >2014-08-25 16:31:11,200 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:11,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1e151475 >2014-08-25 16:31:11,202 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:31:11,452 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:31:11,453 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:31:11,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:11,607 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:11,607 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:11,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 850b175 >2014-08-25 16:31:11,860 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:11,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 850b175 >2014-08-25 16:31:11,896 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:31:11,897 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 16:31:31,896 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-52) Power Management Health Check started. >2014-08-25 16:31:32,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:32,037 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:32,068 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:32,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4d8a068a >2014-08-25 16:31:32,331 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:32,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4d8a068a >2014-08-25 16:31:32,333 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:31:32,465 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:31:32,466 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:31:32,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:32,532 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:32,533 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-52) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:32,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5e3f2f7a >2014-08-25 16:31:32,800 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:32,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5e3f2f7a >2014-08-25 16:31:32,838 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:31:32,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-52) Power Management Health Check completed. >2014-08-25 16:31:52,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check started. >2014-08-25 16:31:53,007 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:53,008 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:53,045 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:53,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3f502335 >2014-08-25 16:31:53,274 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:53,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3f502335 >2014-08-25 16:31:53,277 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:31:53,402 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:31:53,403 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:31:53,467 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:31:53,469 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:31:53,470 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:31:53,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 72f4a8f2 >2014-08-25 16:31:53,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:31:53,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 72f4a8f2 >2014-08-25 16:31:53,733 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:31:53,734 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check completed. >2014-08-25 16:32:13,736 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check started. >2014-08-25 16:32:13,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:13,869 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:13,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:13,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 74e03c01 >2014-08-25 16:32:14,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:14,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 74e03c01 >2014-08-25 16:32:14,800 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:32:14,962 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:32:14,963 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:32:15,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:15,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:15,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:15,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6813323e >2014-08-25 16:32:15,264 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:15,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6813323e >2014-08-25 16:32:15,300 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:32:15,302 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check completed. >2014-08-25 16:32:35,302 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check started. >2014-08-25 16:32:35,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:35,532 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:35,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:35,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 78788046 >2014-08-25 16:32:35,870 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:35,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 78788046 >2014-08-25 16:32:35,872 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:32:36,044 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:32:36,045 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:32:36,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:36,113 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:36,114 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:36,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1593f36 >2014-08-25 16:32:36,339 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:36,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1593f36 >2014-08-25 16:32:36,375 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:32:36,377 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check completed. >2014-08-25 16:32:56,377 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-23) Power Management Health Check started. >2014-08-25 16:32:56,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:56,509 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:56,541 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:56,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-23) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 49b60fb3 >2014-08-25 16:32:57,270 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:57,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 49b60fb3 >2014-08-25 16:32:57,273 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:32:57,434 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:32:57,435 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:32:57,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:32:57,591 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:32:57,592 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-23) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:32:57,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-23) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 257a8776 >2014-08-25 16:32:57,875 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:32:57,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 257a8776 >2014-08-25 16:32:57,944 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:32:57,945 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-23) Power Management Health Check completed. >2014-08-25 16:33:17,945 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 16:33:18,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:33:18,081 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:33:18,112 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:33:18,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: c4732aa >2014-08-25 16:33:18,351 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:33:18,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: c4732aa >2014-08-25 16:33:18,353 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:33:18,518 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:33:18,519 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:33:18,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:33:18,593 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:33:18,594 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:33:18,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 58a7bb7 >2014-08-25 16:33:18,930 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:33:18,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 58a7bb7 >2014-08-25 16:33:18,966 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:33:18,967 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 16:33:38,969 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check started. >2014-08-25 16:33:39,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:33:39,096 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:33:39,126 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:33:39,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3a6891e4 >2014-08-25 16:33:39,362 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:33:39,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3a6891e4 >2014-08-25 16:33:39,364 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:33:39,489 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:33:39,490 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:33:39,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:33:39,554 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:33:39,555 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-70) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:33:39,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 184fa24f >2014-08-25 16:33:39,914 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:33:39,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 184fa24f >2014-08-25 16:33:39,954 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:33:39,955 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-70) Power Management Health Check completed. >2014-08-25 16:33:59,956 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check started. >2014-08-25 16:34:00,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:00,085 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:00,118 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:00,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1fd09b3a >2014-08-25 16:34:00,346 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:00,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1fd09b3a >2014-08-25 16:34:00,348 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:34:00,480 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:34:00,481 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:34:00,566 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:00,567 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:00,568 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:00,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5269c52c >2014-08-25 16:34:00,799 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:00,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5269c52c >2014-08-25 16:34:00,837 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:34:00,838 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check completed. >2014-08-25 16:34:20,840 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check started. >2014-08-25 16:34:20,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:20,969 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:21,001 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:21,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: d5c758c >2014-08-25 16:34:21,399 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:21,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: d5c758c >2014-08-25 16:34:21,401 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:34:21,528 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:34:21,529 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:34:21,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:21,644 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:21,645 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:21,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7a26e711 >2014-08-25 16:34:21,876 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:21,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7a26e711 >2014-08-25 16:34:21,957 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:34:21,958 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check completed. >2014-08-25 16:34:41,960 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-91) Power Management Health Check started. >2014-08-25 16:34:42,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:42,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:42,119 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:42,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 56b82c81 >2014-08-25 16:34:42,485 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:42,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 56b82c81 >2014-08-25 16:34:42,487 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:34:42,613 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:34:42,613 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:34:42,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:34:42,681 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:34:42,682 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-91) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:34:42,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 22e19969 >2014-08-25 16:34:43,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:34:43,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 22e19969 >2014-08-25 16:34:43,102 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:34:43,104 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-91) Power Management Health Check completed. >2014-08-25 16:35:03,104 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check started. >2014-08-25 16:35:03,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:03,232 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:03,264 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:03,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3792d734 >2014-08-25 16:35:03,535 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:03,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3792d734 >2014-08-25 16:35:03,537 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:35:03,716 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:35:03,717 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:35:03,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:03,787 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:03,787 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:03,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6130af76 >2014-08-25 16:35:04,090 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:04,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6130af76 >2014-08-25 16:35:04,216 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:35:04,217 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check completed. >2014-08-25 16:35:24,218 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check started. >2014-08-25 16:35:24,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:24,348 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:24,383 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:24,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 26ff699b >2014-08-25 16:35:24,614 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:24,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 26ff699b >2014-08-25 16:35:24,616 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:35:24,747 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:35:24,747 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:35:24,814 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:24,815 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:24,816 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:24,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50ce00b2 >2014-08-25 16:35:25,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:25,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50ce00b2 >2014-08-25 16:35:25,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:35:25,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check completed. >2014-08-25 16:35:45,080 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check started. >2014-08-25 16:35:45,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:45,209 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:45,240 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:45,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 66755976 >2014-08-25 16:35:45,531 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:45,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 66755976 >2014-08-25 16:35:45,533 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:35:45,660 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:35:45,661 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:35:45,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:35:45,759 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:35:45,760 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:35:45,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 56065f34 >2014-08-25 16:35:46,032 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:35:46,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 56065f34 >2014-08-25 16:35:46,071 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:35:46,072 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check completed. >2014-08-25 16:36:06,072 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check started. >2014-08-25 16:36:06,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:06,202 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:06,233 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:06,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6425bb51 >2014-08-25 16:36:06,630 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:06,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6425bb51 >2014-08-25 16:36:06,633 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:36:06,762 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:36:06,763 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:36:06,830 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:06,831 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:06,832 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:06,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 29b66707 >2014-08-25 16:36:07,062 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:07,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 29b66707 >2014-08-25 16:36:07,099 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:36:07,100 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check completed. >2014-08-25 16:36:27,100 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check started. >2014-08-25 16:36:27,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:27,259 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:27,298 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:27,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 61a4fcf0 >2014-08-25 16:36:27,571 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:27,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 61a4fcf0 >2014-08-25 16:36:27,574 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:36:27,702 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:36:27,703 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:36:27,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:27,772 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:27,773 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-26) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:27,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4fb3dbf5 >2014-08-25 16:36:28,111 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:28,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4fb3dbf5 >2014-08-25 16:36:28,145 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:36:28,147 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-26) Power Management Health Check completed. >2014-08-25 16:36:48,148 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) Power Management Health Check started. >2014-08-25 16:36:48,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:48,274 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:48,305 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:48,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: fd319c >2014-08-25 16:36:48,537 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:48,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: fd319c >2014-08-25 16:36:48,540 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:36:48,668 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:36:48,669 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:36:48,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:36:48,769 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:36:48,770 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:36:48,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 39a972c4 >2014-08-25 16:36:49,005 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:36:49,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 39a972c4 >2014-08-25 16:36:49,041 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:36:49,042 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) Power Management Health Check completed. >2014-08-25 16:37:09,043 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check started. >2014-08-25 16:37:09,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:09,172 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:09,202 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:09,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4bb2c856 >2014-08-25 16:37:09,429 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:09,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4bb2c856 >2014-08-25 16:37:09,432 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:37:09,557 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:37:09,558 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:37:09,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:09,625 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:09,626 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:09,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 219f86d6 >2014-08-25 16:37:09,867 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:09,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 219f86d6 >2014-08-25 16:37:09,905 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:37:09,906 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check completed. >2014-08-25 16:37:29,906 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check started. >2014-08-25 16:37:30,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:30,042 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:30,073 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:30,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50bbfb62 >2014-08-25 16:37:30,479 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:30,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50bbfb62 >2014-08-25 16:37:30,482 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:37:30,608 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:37:30,609 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:37:30,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:30,677 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:30,677 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:30,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50bbfc6a >2014-08-25 16:37:30,902 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:30,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50bbfc6a >2014-08-25 16:37:30,938 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:37:30,940 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check completed. >2014-08-25 16:37:50,940 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check started. >2014-08-25 16:37:51,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:51,073 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:51,104 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:51,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1914b8d >2014-08-25 16:37:51,332 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:51,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1914b8d >2014-08-25 16:37:51,335 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:37:51,469 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:37:51,470 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:37:51,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:37:51,536 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:37:51,537 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:37:51,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 183997ca >2014-08-25 16:37:51,767 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:37:51,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 183997ca >2014-08-25 16:37:51,803 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:37:51,804 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check completed. >2014-08-25 16:38:11,805 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check started. >2014-08-25 16:38:11,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:11,934 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:11,966 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:11,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 73bb4449 >2014-08-25 16:38:12,687 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:12,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 73bb4449 >2014-08-25 16:38:12,689 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:38:12,836 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:38:12,837 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:38:12,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:12,976 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:12,977 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:12,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7e15cf1c >2014-08-25 16:38:13,287 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:13,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7e15cf1c >2014-08-25 16:38:13,323 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:38:13,324 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check completed. >2014-08-25 16:38:33,325 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check started. >2014-08-25 16:38:33,455 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:33,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:33,491 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:33,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 384425bd >2014-08-25 16:38:33,815 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:33,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 384425bd >2014-08-25 16:38:33,818 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:38:33,957 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:38:33,957 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:38:34,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:34,094 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:34,095 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-61) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:34,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: d5d3335 >2014-08-25 16:38:34,343 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:34,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: d5d3335 >2014-08-25 16:38:34,378 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:38:34,379 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-61) Power Management Health Check completed. >2014-08-25 16:38:54,380 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check started. >2014-08-25 16:38:54,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:54,510 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:54,543 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:54,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 534d05c2 >2014-08-25 16:38:54,777 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:54,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 534d05c2 >2014-08-25 16:38:54,779 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:38:54,908 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:38:54,908 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:38:55,066 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:38:55,068 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:38:55,069 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-32) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:38:55,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4002b9de >2014-08-25 16:38:55,414 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:38:55,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4002b9de >2014-08-25 16:38:55,450 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:38:55,452 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-32) Power Management Health Check completed. >2014-08-25 16:39:15,452 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 16:39:15,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:15,638 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:15,670 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:15,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 8083a4d >2014-08-25 16:39:16,018 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:16,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 8083a4d >2014-08-25 16:39:16,020 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:39:16,197 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:39:16,198 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:39:16,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:16,267 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:16,268 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:16,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f2eb7c3 >2014-08-25 16:39:16,494 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:16,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f2eb7c3 >2014-08-25 16:39:16,530 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:39:16,532 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 16:39:36,532 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check started. >2014-08-25 16:39:36,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:36,675 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:36,708 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:36,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2da5b3b6 >2014-08-25 16:39:36,937 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:36,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2da5b3b6 >2014-08-25 16:39:36,939 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:39:37,073 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:39:37,074 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:39:37,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:37,147 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:37,147 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:37,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 44e71322 >2014-08-25 16:39:37,383 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:37,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 44e71322 >2014-08-25 16:39:37,418 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:39:37,419 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check completed. >2014-08-25 16:39:57,420 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 16:39:57,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:57,554 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:57,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:57,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 38f74289 >2014-08-25 16:39:57,821 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:57,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 38f74289 >2014-08-25 16:39:57,823 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:39:58,052 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:39:58,053 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:39:58,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:39:58,119 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:39:58,120 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:39:58,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 62fc4bdd >2014-08-25 16:39:58,346 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:39:58,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 62fc4bdd >2014-08-25 16:39:58,381 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:39:58,382 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 16:40:18,382 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-12) Power Management Health Check started. >2014-08-25 16:40:18,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:40:18,509 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:40:18,541 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:40:18,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 610e8fef >2014-08-25 16:40:18,765 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:40:18,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 610e8fef >2014-08-25 16:40:18,767 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:40:18,914 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:40:18,915 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:40:19,001 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:40:19,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:40:19,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:40:19,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4ae5d7e8 >2014-08-25 16:40:19,257 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:40:19,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4ae5d7e8 >2014-08-25 16:40:19,294 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:40:19,295 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-12) Power Management Health Check completed. >2014-08-25 16:40:39,296 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check started. >2014-08-25 16:40:39,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:40:39,426 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:40:39,457 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:40:39,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f29dd15 >2014-08-25 16:40:39,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:40:39,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f29dd15 >2014-08-25 16:40:39,891 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:40:40,065 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:40:40,066 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:40:40,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:40:40,138 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:40:40,138 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:40:40,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7389560f >2014-08-25 16:40:40,363 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:40:40,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7389560f >2014-08-25 16:40:40,398 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:40:40,399 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check completed. >2014-08-25 16:41:00,400 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:41:00,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:00,529 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:00,560 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:00,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 63e4f1c7 >2014-08-25 16:41:00,790 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:00,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 63e4f1c7 >2014-08-25 16:41:00,792 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:41:00,947 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:41:00,948 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:41:01,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:01,014 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:01,015 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:01,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 11bbefc6 >2014-08-25 16:41:01,261 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 11bbefc6 >2014-08-25 16:41:01,325 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:41:01,326 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:41:21,327 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check started. >2014-08-25 16:41:21,455 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:21,456 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:21,488 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:21,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 700f4ede >2014-08-25 16:41:21,717 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:21,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 700f4ede >2014-08-25 16:41:21,720 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:41:21,855 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:41:21,856 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:41:21,922 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:21,924 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:21,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:21,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2de1c0fd >2014-08-25 16:41:22,156 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:22,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2de1c0fd >2014-08-25 16:41:22,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:41:22,193 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check completed. >2014-08-25 16:41:42,193 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 16:41:42,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:42,322 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:42,353 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:42,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 42fb97de >2014-08-25 16:41:42,582 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:42,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 42fb97de >2014-08-25 16:41:42,584 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:41:42,713 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:41:42,714 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:41:42,780 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:41:42,781 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:41:42,783 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:41:42,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2b7356d2 >2014-08-25 16:41:43,013 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:41:43,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2b7356d2 >2014-08-25 16:41:43,049 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:41:43,050 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 16:42:03,051 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-83) Power Management Health Check started. >2014-08-25 16:42:03,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:03,180 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:03,247 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:03,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1471c38a >2014-08-25 16:42:03,482 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:03,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1471c38a >2014-08-25 16:42:03,484 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:42:03,613 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:42:03,614 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:42:03,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:03,679 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:03,680 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:03,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5394e0a1 >2014-08-25 16:42:03,909 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:03,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5394e0a1 >2014-08-25 16:42:03,945 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:42:03,946 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-83) Power Management Health Check completed. >2014-08-25 16:42:23,946 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 16:42:24,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:24,075 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:24,106 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:24,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6bc13808 >2014-08-25 16:42:24,340 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:24,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6bc13808 >2014-08-25 16:42:24,342 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:42:24,469 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:42:24,469 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:42:24,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:24,536 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:24,536 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:24,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 132a101a >2014-08-25 16:42:24,923 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:24,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 132a101a >2014-08-25 16:42:24,959 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:42:24,960 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 16:42:44,960 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-14) Power Management Health Check started. >2014-08-25 16:42:45,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:45,091 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:45,125 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:45,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 121c9f2d >2014-08-25 16:42:45,353 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:45,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 121c9f2d >2014-08-25 16:42:45,356 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:42:45,486 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:42:45,486 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:42:45,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:42:45,554 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:42:45,555 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:42:45,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1964e8c1 >2014-08-25 16:42:45,782 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:42:45,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1964e8c1 >2014-08-25 16:42:45,820 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:42:45,821 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-14) Power Management Health Check completed. >2014-08-25 16:43:05,822 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check started. >2014-08-25 16:43:05,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:05,951 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:05,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:05,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57d44014 >2014-08-25 16:43:06,212 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:06,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57d44014 >2014-08-25 16:43:06,214 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:43:06,341 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:43:06,342 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:43:06,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:06,409 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:06,410 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-75) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:06,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ae3eeb4 >2014-08-25 16:43:06,635 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:06,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ae3eeb4 >2014-08-25 16:43:06,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:43:06,671 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-75) Power Management Health Check completed. >2014-08-25 16:43:26,671 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-54) Power Management Health Check started. >2014-08-25 16:43:26,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:26,801 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:26,831 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:26,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 17927e48 >2014-08-25 16:43:27,560 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:27,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 17927e48 >2014-08-25 16:43:27,562 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:43:27,690 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:43:27,691 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:43:27,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:27,757 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:27,758 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:27,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 734f8dfd >2014-08-25 16:43:28,052 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:28,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 734f8dfd >2014-08-25 16:43:28,089 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:43:28,090 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-54) Power Management Health Check completed. >2014-08-25 16:43:48,090 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-28) Power Management Health Check started. >2014-08-25 16:43:48,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:48,218 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:48,248 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:48,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3dded596 >2014-08-25 16:43:48,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:48,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3dded596 >2014-08-25 16:43:48,700 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:43:48,849 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:43:48,850 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:43:48,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:43:48,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:43:48,919 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:43:48,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 291f6e53 >2014-08-25 16:43:49,152 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:43:49,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 291f6e53 >2014-08-25 16:43:49,188 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:43:49,190 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-28) Power Management Health Check completed. >2014-08-25 16:44:09,190 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check started. >2014-08-25 16:44:09,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:09,323 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:09,353 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:09,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57228db9 >2014-08-25 16:44:09,578 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:09,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57228db9 >2014-08-25 16:44:09,580 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:44:09,743 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:44:09,744 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:44:09,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:09,817 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:09,818 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:09,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5601d6f1 >2014-08-25 16:44:10,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:10,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5601d6f1 >2014-08-25 16:44:10,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:44:10,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-87) Power Management Health Check completed. >2014-08-25 16:44:30,081 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check started. >2014-08-25 16:44:30,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:30,210 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:30,241 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:30,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3d1109d2 >2014-08-25 16:44:30,484 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:30,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3d1109d2 >2014-08-25 16:44:30,486 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:44:30,654 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:44:30,655 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:44:30,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:30,721 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:30,722 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:30,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 49e7f976 >2014-08-25 16:44:30,953 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:30,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 49e7f976 >2014-08-25 16:44:30,993 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:44:30,995 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check completed. >2014-08-25 16:44:33,917 INFO [org.ovirt.engine.core.bll.aaa.LoginBaseCommand] (ajp--127.0.0.1-8702-7) Cant login user "admin" with authentication profile "internal" because the authentication failed. >2014-08-25 16:44:33,933 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin cannot login, please verify the username and password. >2014-08-25 16:44:33,935 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin failed to log in. >2014-08-25 16:44:33,936 WARN [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-7) CanDoAction of action LoginAdminUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD >2014-08-25 16:44:39,494 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-08-25 16:44:39,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-25 16:44:39,626 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-25 16:44:50,997 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-38) Power Management Health Check started. >2014-08-25 16:44:51,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:51,270 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:51,358 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:51,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 14edac49 >2014-08-25 16:44:51,842 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:51,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 14edac49 >2014-08-25 16:44:51,845 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:44:52,216 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:44:52,217 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:44:52,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:44:52,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:44:52,440 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-38) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:44:52,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 785cef92 >2014-08-25 16:44:52,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:44:52,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 785cef92 >2014-08-25 16:44:52,706 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:44:52,707 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-38) Power Management Health Check completed. >2014-08-25 16:45:12,707 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) Power Management Health Check started. >2014-08-25 16:45:12,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:12,928 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:12,978 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:12,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 17015abb >2014-08-25 16:45:13,380 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:13,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 17015abb >2014-08-25 16:45:13,382 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:45:13,510 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:45:13,511 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:45:13,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:13,577 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:13,578 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:13,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6eb1079f >2014-08-25 16:45:13,808 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:13,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6eb1079f >2014-08-25 16:45:13,843 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:45:13,844 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-10) Power Management Health Check completed. >2014-08-25 16:45:33,844 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 16:45:34,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:34,139 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:34,214 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:34,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5f345444 >2014-08-25 16:45:34,688 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:34,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5f345444 >2014-08-25 16:45:34,691 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:45:34,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:34,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 16:45:34,830 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:34,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2b6cb807 >2014-08-25 16:45:35,208 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:45:35,209 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:45:35,313 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:35,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2b6cb807 >2014-08-25 16:45:35,315 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:45:35,507 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:35,509 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:35,509 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:35,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6d43e3fb >2014-08-25 16:45:35,797 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Failed to run Power Management command on Host , no running proxy Host was found. >2014-08-25 16:45:35,798 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:45:35,961 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:35,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6d43e3fb >2014-08-25 16:45:36,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:36,081 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-25 16:45:36,082 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-11) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:36,098 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:45:36,099 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 16:45:36,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-11) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 196e6869 >2014-08-25 16:45:36,368 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:36,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-11) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 196e6869 >2014-08-25 16:45:56,100 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check started. >2014-08-25 16:45:56,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:56,282 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:56,329 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:56,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 557e7201 >2014-08-25 16:45:56,563 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:56,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 557e7201 >2014-08-25 16:45:56,565 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:45:56,703 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:45:56,704 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:45:56,771 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:45:56,773 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:45:56,774 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:45:56,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f0763c2 >2014-08-25 16:45:56,999 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:45:57,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f0763c2 >2014-08-25 16:45:57,035 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:45:57,036 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-43) Power Management Health Check completed. >2014-08-25 16:46:17,037 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check started. >2014-08-25 16:46:17,165 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:17,167 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:17,201 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:17,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 758f738c >2014-08-25 16:46:17,427 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:17,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 758f738c >2014-08-25 16:46:17,429 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:46:17,555 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:46:17,556 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:46:17,663 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:17,665 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:17,666 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:17,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2d4195ff >2014-08-25 16:46:17,986 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:17,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2d4195ff >2014-08-25 16:46:18,022 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:46:18,024 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check completed. >2014-08-25 16:46:38,024 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check started. >2014-08-25 16:46:38,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:38,154 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:38,185 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:38,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 77cf0607 >2014-08-25 16:46:38,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:38,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 77cf0607 >2014-08-25 16:46:38,413 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:46:38,539 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:46:38,540 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:46:38,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:38,606 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:38,607 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:38,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 233b999d >2014-08-25 16:46:38,953 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:38,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 233b999d >2014-08-25 16:46:38,988 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:46:38,990 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check completed. >2014-08-25 16:46:58,991 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:46:59,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:59,121 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:59,152 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:59,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 42a1b4e5 >2014-08-25 16:46:59,380 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:59,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 42a1b4e5 >2014-08-25 16:46:59,382 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:46:59,520 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:46:59,520 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:46:59,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:46:59,586 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:46:59,586 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:46:59,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 52103b6f >2014-08-25 16:46:59,810 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:46:59,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 52103b6f >2014-08-25 16:46:59,845 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:46:59,846 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:47:19,847 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check started. >2014-08-25 16:47:19,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:47:19,977 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:47:20,008 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:47:20,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 74a32ce8 >2014-08-25 16:47:20,242 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:47:20,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 74a32ce8 >2014-08-25 16:47:20,245 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:47:20,427 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:47:20,428 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:47:20,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:47:20,592 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:47:20,593 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:47:20,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4e76590c >2014-08-25 16:47:20,944 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:47:20,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4e76590c >2014-08-25 16:47:21,013 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:47:21,014 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check completed. >2014-08-25 16:47:41,015 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 16:47:41,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:47:41,145 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:47:41,175 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:47:41,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4c46b4be >2014-08-25 16:47:41,405 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:47:41,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4c46b4be >2014-08-25 16:47:41,407 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:47:41,532 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:47:41,533 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:47:41,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:47:41,600 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:47:41,601 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:47:41,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 24128dc5 >2014-08-25 16:47:41,829 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:47:41,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 24128dc5 >2014-08-25 16:47:41,866 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:47:41,868 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 16:47:54,378 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-25 16:48:01,868 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check started. >2014-08-25 16:48:02,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:02,005 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:02,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1128a9d0 >2014-08-25 16:48:02,261 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1128a9d0 >2014-08-25 16:48:02,263 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:48:02,391 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:48:02,392 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:48:02,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:02,457 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:02,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:02,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 556fc791 >2014-08-25 16:48:02,685 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:02,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 556fc791 >2014-08-25 16:48:02,720 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:48:02,721 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check completed. >2014-08-25 16:48:22,722 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-49) Power Management Health Check started. >2014-08-25 16:48:22,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:22,856 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:22,887 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:22,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3f7fb47f >2014-08-25 16:48:23,161 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:23,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3f7fb47f >2014-08-25 16:48:23,164 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:48:23,370 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:48:23,375 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:48:23,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:23,442 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:23,443 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:23,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4957d513 >2014-08-25 16:48:23,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:23,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4957d513 >2014-08-25 16:48:23,756 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:48:23,757 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-49) Power Management Health Check completed. >2014-08-25 16:48:43,758 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-12) Power Management Health Check started. >2014-08-25 16:48:43,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:43,889 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:43,921 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:43,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1ef8414f >2014-08-25 16:48:44,640 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:44,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1ef8414f >2014-08-25 16:48:44,642 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:48:44,804 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:48:44,805 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:48:44,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:48:44,870 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:48:44,870 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:48:44,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5bf75198 >2014-08-25 16:48:45,182 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:48:45,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5bf75198 >2014-08-25 16:48:45,222 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:48:45,223 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-12) Power Management Health Check completed. >2014-08-25 16:49:05,223 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-89) Power Management Health Check started. >2014-08-25 16:49:05,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:05,387 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:05,460 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:05,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5e2027a8 >2014-08-25 16:49:05,757 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:05,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5e2027a8 >2014-08-25 16:49:05,760 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:49:05,989 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:49:05,990 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:49:06,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:06,057 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:06,058 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:06,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: a8a4b20 >2014-08-25 16:49:06,295 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:06,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: a8a4b20 >2014-08-25 16:49:06,331 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:49:06,333 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-89) Power Management Health Check completed. >2014-08-25 16:49:26,333 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check started. >2014-08-25 16:49:26,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:26,464 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:26,495 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:26,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 102216a4 >2014-08-25 16:49:26,720 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:26,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 102216a4 >2014-08-25 16:49:26,722 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:49:26,850 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:49:26,851 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:49:26,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:26,917 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:26,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:26,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6fcacd21 >2014-08-25 16:49:27,144 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:27,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6fcacd21 >2014-08-25 16:49:27,180 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:49:27,181 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check completed. >2014-08-25 16:49:47,183 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 16:49:47,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:47,312 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:47,345 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:47,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5dc27dd7 >2014-08-25 16:49:47,572 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:47,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5dc27dd7 >2014-08-25 16:49:47,574 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:49:47,700 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:49:47,701 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:49:47,766 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:49:47,768 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:49:47,768 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:49:47,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7ff08744 >2014-08-25 16:49:47,998 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:49:47,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7ff08744 >2014-08-25 16:49:48,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:49:48,035 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 16:50:08,037 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check started. >2014-08-25 16:50:08,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:08,164 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:08,195 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:08,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 39b0734a >2014-08-25 16:50:08,421 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:08,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 39b0734a >2014-08-25 16:50:08,424 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:50:08,551 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:50:08,552 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:50:08,618 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:08,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:08,620 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:08,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 430e337f >2014-08-25 16:50:08,848 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:08,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 430e337f >2014-08-25 16:50:08,916 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:50:08,918 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check completed. >2014-08-25 16:50:28,918 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check started. >2014-08-25 16:50:29,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:29,051 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:29,085 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:29,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3f48d148 >2014-08-25 16:50:29,312 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:29,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3f48d148 >2014-08-25 16:50:29,314 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:50:29,443 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:50:29,444 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:50:29,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:29,514 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:29,515 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:29,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57e45086 >2014-08-25 16:50:29,740 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:29,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57e45086 >2014-08-25 16:50:29,785 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:50:29,787 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check completed. >2014-08-25 16:50:49,787 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-40) Power Management Health Check started. >2014-08-25 16:50:50,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:50,150 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:50,181 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:50,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7529ba67 >2014-08-25 16:50:50,406 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:50,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7529ba67 >2014-08-25 16:50:50,408 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:50:50,534 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:50:50,535 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:50:50,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:50:50,602 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:50:50,603 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:50:50,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 46281245 >2014-08-25 16:50:50,827 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:50:50,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 46281245 >2014-08-25 16:50:50,863 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:50:50,864 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-40) Power Management Health Check completed. >2014-08-25 16:51:10,865 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-19) Power Management Health Check started. >2014-08-25 16:51:10,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:10,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:11,020 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:11,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 52b05beb >2014-08-25 16:51:11,250 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:11,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 52b05beb >2014-08-25 16:51:11,253 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:51:11,382 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:51:11,383 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:51:11,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:11,481 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:11,482 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:11,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 670efce5 >2014-08-25 16:51:11,771 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:11,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 670efce5 >2014-08-25 16:51:11,840 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:51:11,841 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-19) Power Management Health Check completed. >2014-08-25 16:51:31,842 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 16:51:31,978 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:31,979 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:32,011 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:32,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 598d1176 >2014-08-25 16:51:32,277 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:32,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 598d1176 >2014-08-25 16:51:32,280 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:51:32,409 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:51:32,410 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:51:32,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:32,483 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:32,484 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:32,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 41a8dce0 >2014-08-25 16:51:32,848 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:32,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 41a8dce0 >2014-08-25 16:51:32,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:51:32,889 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 16:51:52,890 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-46) Power Management Health Check started. >2014-08-25 16:51:53,017 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:53,018 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:53,049 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:53,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e6cc7b1 >2014-08-25 16:51:53,276 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:53,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e6cc7b1 >2014-08-25 16:51:53,278 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:51:53,405 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:51:53,406 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:51:53,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:51:53,471 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:51:53,473 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:51:53,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 69a38a90 >2014-08-25 16:51:53,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:51:53,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 69a38a90 >2014-08-25 16:51:53,832 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:51:53,834 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-46) Power Management Health Check completed. >2014-08-25 16:52:13,834 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check started. >2014-08-25 16:52:13,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:13,963 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:13,995 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:13,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f512afc >2014-08-25 16:52:14,491 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:14,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f512afc >2014-08-25 16:52:14,493 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:52:14,624 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:52:14,625 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:52:14,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:14,691 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:14,691 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-17) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:14,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 475afd87 >2014-08-25 16:52:15,057 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:15,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 475afd87 >2014-08-25 16:52:15,093 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:52:15,095 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-17) Power Management Health Check completed. >2014-08-25 16:52:35,095 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check started. >2014-08-25 16:52:35,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:35,225 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:35,255 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:35,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7bddf5b9 >2014-08-25 16:52:35,484 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:35,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7bddf5b9 >2014-08-25 16:52:35,487 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:52:35,619 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:52:35,619 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:52:35,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:35,685 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:35,686 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:35,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1b1f1068 >2014-08-25 16:52:35,922 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:35,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1b1f1068 >2014-08-25 16:52:35,961 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:52:35,962 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check completed. >2014-08-25 16:52:55,963 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:52:56,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:56,104 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:56,146 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:56,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 27179dc3 >2014-08-25 16:52:56,372 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:56,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 27179dc3 >2014-08-25 16:52:56,374 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:52:56,500 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:52:56,501 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:52:56,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:52:56,604 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:52:56,605 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:52:56,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7290f2c7 >2014-08-25 16:52:56,834 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:52:56,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7290f2c7 >2014-08-25 16:52:56,870 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:52:56,872 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:53:16,872 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check started. >2014-08-25 16:53:17,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:17,004 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:17,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:17,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2bcdba6 >2014-08-25 16:53:17,263 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:53:17,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2bcdba6 >2014-08-25 16:53:17,266 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:53:17,402 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:53:17,403 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:53:17,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:17,469 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:17,470 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-31) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:17,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4e10a563 >2014-08-25 16:53:17,696 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:53:17,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4e10a563 >2014-08-25 16:53:17,732 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:53:17,733 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-31) Power Management Health Check completed. >2014-08-25 16:53:37,735 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 16:53:37,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:37,862 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:37,893 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:37,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 33061926 >2014-08-25 16:53:38,169 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:53:38,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 33061926 >2014-08-25 16:53:38,171 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:53:38,361 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:53:38,362 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:53:38,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:38,432 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:38,433 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:38,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3714ca0 >2014-08-25 16:53:38,668 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:53:38,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3714ca0 >2014-08-25 16:53:38,704 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:53:38,705 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 16:53:58,706 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-74) Power Management Health Check started. >2014-08-25 16:53:58,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:58,838 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:58,868 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:58,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 678a6e1f >2014-08-25 16:53:59,631 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:53:59,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 678a6e1f >2014-08-25 16:53:59,634 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:53:59,785 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:53:59,785 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:53:59,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:53:59,865 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:53:59,865 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-74) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:53:59,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6c1df900 >2014-08-25 16:54:00,091 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:54:00,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6c1df900 >2014-08-25 16:54:00,127 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:54:00,128 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-74) Power Management Health Check completed. >2014-08-25 16:54:20,129 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-42) Power Management Health Check started. >2014-08-25 16:54:20,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:54:20,258 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:54:20,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:54:20,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1b05b1ad >2014-08-25 16:54:20,522 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:54:20,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1b05b1ad >2014-08-25 16:54:20,524 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:54:20,654 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:54:20,655 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:54:20,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:54:20,721 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:54:20,722 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-42) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:54:20,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 690dc68f >2014-08-25 16:54:20,954 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:54:20,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 690dc68f >2014-08-25 16:54:20,990 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:54:20,991 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-42) Power Management Health Check completed. >2014-08-25 16:54:40,991 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check started. >2014-08-25 16:54:41,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:54:41,122 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:54:41,153 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:54:41,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4ad77352 >2014-08-25 16:54:41,379 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:54:41,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4ad77352 >2014-08-25 16:54:41,381 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:54:41,509 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:54:41,509 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:54:41,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:54:41,577 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:54:41,578 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:54:41,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7e9311ed >2014-08-25 16:54:41,802 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:54:41,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7e9311ed >2014-08-25 16:54:41,841 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:54:41,843 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check completed. >2014-08-25 16:55:01,844 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-97) Power Management Health Check started. >2014-08-25 16:55:01,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:01,970 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:02,000 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:02,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 785fc6e4 >2014-08-25 16:55:02,227 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:02,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 785fc6e4 >2014-08-25 16:55:02,230 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:55:02,360 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:55:02,361 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:55:02,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:02,429 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:02,430 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-97) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:02,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2bfbb9c5 >2014-08-25 16:55:02,660 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:02,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2bfbb9c5 >2014-08-25 16:55:02,696 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:55:02,697 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-97) Power Management Health Check completed. >2014-08-25 16:55:22,700 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check started. >2014-08-25 16:55:22,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:22,830 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:22,862 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:22,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 37db10a >2014-08-25 16:55:23,095 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:23,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 37db10a >2014-08-25 16:55:23,098 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:55:23,297 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:55:23,298 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:55:23,370 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:23,371 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:23,372 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:23,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 732337aa >2014-08-25 16:55:23,712 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:23,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 732337aa >2014-08-25 16:55:23,749 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:55:23,750 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check completed. >2014-08-25 16:55:43,755 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check started. >2014-08-25 16:55:43,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:43,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:43,932 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:43,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 71a005c6 >2014-08-25 16:55:44,186 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:44,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 71a005c6 >2014-08-25 16:55:44,189 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:55:44,319 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:55:44,320 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:55:44,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:55:44,388 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:55:44,389 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:55:44,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 59e5fb6e >2014-08-25 16:55:44,618 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:55:44,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 59e5fb6e >2014-08-25 16:55:44,654 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:55:44,655 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check completed. >2014-08-25 16:56:04,657 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 16:56:04,781 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:04,783 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:04,817 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:04,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5d213ebc >2014-08-25 16:56:05,047 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:05,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5d213ebc >2014-08-25 16:56:05,049 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:56:05,199 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:56:05,200 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:56:05,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:05,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:05,290 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:05,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7d536d39 >2014-08-25 16:56:05,514 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:05,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7d536d39 >2014-08-25 16:56:05,582 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:56:05,583 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 16:56:25,583 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check started. >2014-08-25 16:56:25,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:25,733 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:25,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:25,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4821da06 >2014-08-25 16:56:25,993 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:25,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4821da06 >2014-08-25 16:56:25,996 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:56:26,129 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:56:26,130 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:56:26,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:26,305 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:26,306 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:26,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: ee2554d >2014-08-25 16:56:26,532 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:26,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: ee2554d >2014-08-25 16:56:26,580 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:56:26,581 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check completed. >2014-08-25 16:56:46,583 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-48) Power Management Health Check started. >2014-08-25 16:56:46,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:46,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:46,784 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:46,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 18a3277a >2014-08-25 16:56:47,074 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:47,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 18a3277a >2014-08-25 16:56:47,076 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:56:47,206 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:56:47,207 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:56:47,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:56:47,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:56:47,290 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:56:47,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3401ee31 >2014-08-25 16:56:47,562 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:56:47,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3401ee31 >2014-08-25 16:56:47,599 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:56:47,601 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-48) Power Management Health Check completed. >2014-08-25 16:57:07,602 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check started. >2014-08-25 16:57:07,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:07,733 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:07,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:07,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c4054da >2014-08-25 16:57:07,995 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:07,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c4054da >2014-08-25 16:57:07,997 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:57:08,128 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:57:08,129 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:57:08,195 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:08,196 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:08,197 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:08,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 89f375 >2014-08-25 16:57:08,616 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:08,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 89f375 >2014-08-25 16:57:08,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:57:08,660 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check completed. >2014-08-25 16:57:28,661 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check started. >2014-08-25 16:57:28,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:28,790 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:28,822 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:28,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2581880b >2014-08-25 16:57:29,052 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:29,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2581880b >2014-08-25 16:57:29,054 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:57:29,185 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:57:29,186 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:57:29,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:29,252 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:29,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-73) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:29,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 53f14083 >2014-08-25 16:57:29,477 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:29,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 53f14083 >2014-08-25 16:57:29,514 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:57:29,515 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-73) Power Management Health Check completed. >2014-08-25 16:57:49,517 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check started. >2014-08-25 16:57:49,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:49,645 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:49,676 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:49,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4623f899 >2014-08-25 16:57:49,985 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:49,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4623f899 >2014-08-25 16:57:49,987 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:57:50,115 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:57:50,116 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:57:50,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:57:50,182 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:57:50,183 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:57:50,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 80f0244 >2014-08-25 16:57:50,423 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:57:50,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 80f0244 >2014-08-25 16:57:50,459 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:57:50,461 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check completed. >2014-08-25 16:58:10,462 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-30) Power Management Health Check started. >2014-08-25 16:58:10,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:10,619 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:10,651 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:10,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 44dcdfed >2014-08-25 16:58:10,886 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:10,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 44dcdfed >2014-08-25 16:58:10,888 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:58:11,124 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:58:11,124 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:58:11,189 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:11,190 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:11,191 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-30) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:11,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 43e4d213 >2014-08-25 16:58:11,519 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:11,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 43e4d213 >2014-08-25 16:58:11,555 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:58:11,556 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-30) Power Management Health Check completed. >2014-08-25 16:58:31,558 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check started. >2014-08-25 16:58:31,687 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:31,688 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:31,720 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:31,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1386554b >2014-08-25 16:58:31,948 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:31,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1386554b >2014-08-25 16:58:31,951 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:58:32,080 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:58:32,081 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:58:32,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:32,155 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:32,156 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:32,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6faee055 >2014-08-25 16:58:32,382 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:32,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6faee055 >2014-08-25 16:58:32,417 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:58:32,418 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check completed. >2014-08-25 16:58:52,420 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check started. >2014-08-25 16:58:52,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:52,551 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:52,583 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:52,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3808467c >2014-08-25 16:58:52,819 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:52,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3808467c >2014-08-25 16:58:52,821 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:58:52,956 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:58:52,957 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:58:53,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:58:53,027 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:58:53,027 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-51) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:58:53,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 61b9e006 >2014-08-25 16:58:53,263 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:58:53,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 61b9e006 >2014-08-25 16:58:53,323 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:58:53,324 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-51) Power Management Health Check completed. >2014-08-25 16:59:13,325 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-40) Power Management Health Check started. >2014-08-25 16:59:13,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:13,453 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:13,485 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:13,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4ce4a483 >2014-08-25 16:59:14,279 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:14,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4ce4a483 >2014-08-25 16:59:14,281 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:59:14,409 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:59:14,410 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:59:14,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:14,477 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:14,478 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-40) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:14,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1adfb332 >2014-08-25 16:59:14,734 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:14,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1adfb332 >2014-08-25 16:59:14,772 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:59:14,774 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-40) Power Management Health Check completed. >2014-08-25 16:59:34,774 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 16:59:34,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:34,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:34,931 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:34,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 70bf2548 >2014-08-25 16:59:35,282 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:35,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 70bf2548 >2014-08-25 16:59:35,285 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:59:35,410 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:59:35,411 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:59:35,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:35,479 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:35,480 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:35,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 573c7f79 >2014-08-25 16:59:35,705 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:35,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 573c7f79 >2014-08-25 16:59:35,741 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:59:35,743 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 16:59:55,744 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-83) Power Management Health Check started. >2014-08-25 16:59:55,869 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:55,870 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:55,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:55,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 37c94f9b >2014-08-25 16:59:56,131 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:56,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 37c94f9b >2014-08-25 16:59:56,133 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 16:59:56,260 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 16:59:56,261 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 16:59:56,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 16:59:56,327 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 16:59:56,328 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-83) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 16:59:56,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 383ba869 >2014-08-25 16:59:56,668 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 16:59:56,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 383ba869 >2014-08-25 16:59:56,706 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 16:59:56,707 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-83) Power Management Health Check completed. >2014-08-25 17:00:16,708 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-55) Power Management Health Check started. >2014-08-25 17:00:16,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:16,837 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:16,868 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:16,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3cf8bc99 >2014-08-25 17:00:17,096 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:17,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3cf8bc99 >2014-08-25 17:00:17,098 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:00:17,228 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:00:17,229 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:00:17,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:17,296 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:17,297 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:17,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 72cd2cc1 >2014-08-25 17:00:17,520 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:17,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 72cd2cc1 >2014-08-25 17:00:17,557 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:00:17,559 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-55) Power Management Health Check completed. >2014-08-25 17:00:37,560 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-19) Power Management Health Check started. >2014-08-25 17:00:37,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:37,689 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:37,720 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:37,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3eb56cfc >2014-08-25 17:00:37,953 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:37,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3eb56cfc >2014-08-25 17:00:37,956 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:00:38,082 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:00:38,083 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:00:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:38,148 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:38,149 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-19) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:38,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 599a31fc >2014-08-25 17:00:38,376 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:38,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 599a31fc >2014-08-25 17:00:38,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:00:38,412 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-19) Power Management Health Check completed. >2014-08-25 17:00:58,413 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check started. >2014-08-25 17:00:58,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:58,543 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:58,574 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:58,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 21088a37 >2014-08-25 17:00:58,913 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:58,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 21088a37 >2014-08-25 17:00:58,916 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:00:59,042 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:00:59,043 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:00:59,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:00:59,110 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:00:59,110 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-81) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:00:59,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6622ff20 >2014-08-25 17:00:59,337 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:00:59,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6622ff20 >2014-08-25 17:00:59,381 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:00:59,382 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-81) Power Management Health Check completed. >2014-08-25 17:01:00,457 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-89) Principal admin::internal synchronized >2014-08-25 17:01:00,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-08-25 17:01:19,383 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check started. >2014-08-25 17:01:19,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:01:19,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:01:19,542 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:01:19,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4cb8cf63 >2014-08-25 17:01:19,766 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:01:19,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4cb8cf63 >2014-08-25 17:01:19,769 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:01:19,918 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:01:19,918 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:01:20,066 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:01:20,068 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:01:20,069 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:01:20,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5c650937 >2014-08-25 17:01:20,295 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:01:20,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5c650937 >2014-08-25 17:01:20,330 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:01:20,331 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check completed. >2014-08-25 17:01:40,332 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check started. >2014-08-25 17:01:40,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:01:40,459 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:01:40,491 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:01:40,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 34706c04 >2014-08-25 17:01:40,717 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:01:40,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 34706c04 >2014-08-25 17:01:40,720 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:01:40,847 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:01:40,848 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:01:40,913 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:01:40,915 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:01:40,916 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-37) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:01:40,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2fc17d3 >2014-08-25 17:01:41,141 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:01:41,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2fc17d3 >2014-08-25 17:01:41,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:01:41,177 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-37) Power Management Health Check completed. >2014-08-25 17:02:01,178 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 17:02:01,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:01,306 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:01,337 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:01,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1923235d >2014-08-25 17:02:01,563 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:01,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1923235d >2014-08-25 17:02:01,566 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:02:01,698 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:02:01,699 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:02:01,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:01,766 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:01,767 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:01,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4b28d9d9 >2014-08-25 17:02:01,994 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:01,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4b28d9d9 >2014-08-25 17:02:02,030 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:02:02,031 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 17:02:22,032 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check started. >2014-08-25 17:02:22,166 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:22,167 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:22,200 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:22,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7a5fea74 >2014-08-25 17:02:22,615 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:22,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7a5fea74 >2014-08-25 17:02:22,617 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:02:22,832 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:02:22,833 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:02:22,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:22,897 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:22,898 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:22,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 150227fe >2014-08-25 17:02:23,189 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:23,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 150227fe >2014-08-25 17:02:23,225 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:02:23,226 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check completed. >2014-08-25 17:02:43,227 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check started. >2014-08-25 17:02:43,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:43,441 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:43,483 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:43,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 50c9ed72 >2014-08-25 17:02:43,847 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:43,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 50c9ed72 >2014-08-25 17:02:43,850 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:02:43,975 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:02:43,976 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:02:44,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:02:44,041 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:02:44,042 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-33) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:02:44,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5db2d1b9 >2014-08-25 17:02:44,277 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:02:44,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5db2d1b9 >2014-08-25 17:02:44,313 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:02:44,315 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-33) Power Management Health Check completed. >2014-08-25 17:03:04,315 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-28) Power Management Health Check started. >2014-08-25 17:03:04,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:04,444 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:04,475 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:04,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3be73a57 >2014-08-25 17:03:04,700 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:04,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3be73a57 >2014-08-25 17:03:04,703 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:03:04,833 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:03:04,834 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:03:04,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:04,900 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:04,901 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-28) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:04,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5f5749e1 >2014-08-25 17:03:05,123 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:05,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5f5749e1 >2014-08-25 17:03:05,159 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:03:05,160 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-28) Power Management Health Check completed. >2014-08-25 17:03:25,161 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-92) Power Management Health Check started. >2014-08-25 17:03:25,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:25,336 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:25,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:25,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 18eef736 >2014-08-25 17:03:25,598 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:25,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 18eef736 >2014-08-25 17:03:25,601 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:03:25,775 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:03:25,776 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:03:25,840 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:25,842 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:25,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:25,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5df161fb >2014-08-25 17:03:26,072 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:26,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5df161fb >2014-08-25 17:03:26,110 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:03:26,112 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-92) Power Management Health Check completed. >2014-08-25 17:03:46,112 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check started. >2014-08-25 17:03:46,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:46,239 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:46,271 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:46,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 415baac >2014-08-25 17:03:46,495 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:46,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 415baac >2014-08-25 17:03:46,497 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:03:46,625 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:03:46,626 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:03:46,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:03:46,693 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:03:46,694 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-56) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:03:46,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1f398ede >2014-08-25 17:03:46,945 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:03:46,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1f398ede >2014-08-25 17:03:46,989 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:03:46,990 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-56) Power Management Health Check completed. >2014-08-25 17:04:06,992 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check started. >2014-08-25 17:04:07,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:07,123 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:07,154 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:07,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 30a0e0d2 >2014-08-25 17:04:07,491 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:07,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 30a0e0d2 >2014-08-25 17:04:07,493 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:04:07,624 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:04:07,625 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:04:07,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:07,695 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:07,696 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-27) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:07,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3ebe46ee >2014-08-25 17:04:07,920 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:07,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3ebe46ee >2014-08-25 17:04:07,956 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:04:07,957 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-27) Power Management Health Check completed. >2014-08-25 17:04:27,958 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check started. >2014-08-25 17:04:28,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:28,091 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:28,121 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:28,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5969c942 >2014-08-25 17:04:28,840 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:28,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5969c942 >2014-08-25 17:04:28,842 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:04:28,967 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:04:28,968 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:04:29,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:29,034 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:29,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-85) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:29,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 58f0295d >2014-08-25 17:04:29,258 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:29,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 58f0295d >2014-08-25 17:04:29,301 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:04:29,302 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-85) Power Management Health Check completed. >2014-08-25 17:04:49,303 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check started. >2014-08-25 17:04:49,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:49,433 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:49,463 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:49,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 330c8e5b >2014-08-25 17:04:49,687 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:49,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 330c8e5b >2014-08-25 17:04:49,689 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:04:49,816 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:04:49,817 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:04:49,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:04:49,882 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:04:49,883 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:04:49,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 613dc0fd >2014-08-25 17:04:50,106 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:04:50,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 613dc0fd >2014-08-25 17:04:50,142 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:04:50,143 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-59) Power Management Health Check completed. >2014-08-25 17:05:10,144 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 17:05:10,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:10,271 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:10,302 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:10,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 53af0807 >2014-08-25 17:05:10,525 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:10,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 53af0807 >2014-08-25 17:05:10,528 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:05:10,653 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:05:10,654 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:05:10,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:10,720 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:10,721 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:10,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 76666c60 >2014-08-25 17:05:10,946 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:10,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 76666c60 >2014-08-25 17:05:10,982 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:05:10,983 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 17:05:30,983 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 17:05:31,161 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:31,162 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:31,215 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:31,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2303bf9f >2014-08-25 17:05:31,500 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:31,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2303bf9f >2014-08-25 17:05:31,503 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:05:31,802 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:05:31,802 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:05:31,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:31,968 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:31,968 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:31,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 52832e3b >2014-08-25 17:05:32,251 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:32,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 52832e3b >2014-08-25 17:05:32,323 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:05:32,325 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 17:05:52,325 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check started. >2014-08-25 17:05:52,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:52,477 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:52,514 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:52,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 35b5078c >2014-08-25 17:05:52,739 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:52,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 35b5078c >2014-08-25 17:05:52,741 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:05:52,871 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:05:52,872 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:05:52,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:05:52,939 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:05:52,940 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-67) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:05:52,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2eb124dc >2014-08-25 17:05:53,165 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:05:53,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2eb124dc >2014-08-25 17:05:53,202 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:05:53,203 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-67) Power Management Health Check completed. >2014-08-25 17:06:13,204 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-63) Power Management Health Check started. >2014-08-25 17:06:13,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:13,332 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:13,365 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:13,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5110ffd1 >2014-08-25 17:06:13,588 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:13,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5110ffd1 >2014-08-25 17:06:13,591 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:06:13,729 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:06:13,729 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:06:13,793 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:13,794 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:13,795 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-63) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:13,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 21cea262 >2014-08-25 17:06:14,030 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:14,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 21cea262 >2014-08-25 17:06:14,066 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:06:14,068 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-63) Power Management Health Check completed. >2014-08-25 17:06:34,068 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-25) [534aca73] Power Management Health Check started. >2014-08-25 17:06:34,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:34,193 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:34,224 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:34,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 42295394 >2014-08-25 17:06:34,466 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:34,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 42295394 >2014-08-25 17:06:34,468 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:06:34,606 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:06:34,607 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:06:34,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:34,675 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:34,676 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-25) [534aca73] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:34,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 45f306da >2014-08-25 17:06:34,900 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:34,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [534aca73] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 45f306da >2014-08-25 17:06:34,936 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [534aca73] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:06:34,938 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-25) [534aca73] Power Management Health Check completed. >2014-08-25 17:06:54,938 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 17:06:55,066 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:55,067 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:55,099 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:55,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79bdd861 >2014-08-25 17:06:55,322 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:55,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79bdd861 >2014-08-25 17:06:55,324 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:06:55,452 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:06:55,453 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:06:55,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:06:55,520 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:06:55,521 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:06:55,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 9f9d7fe >2014-08-25 17:06:55,748 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:06:55,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 9f9d7fe >2014-08-25 17:06:55,784 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:06:55,785 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 17:07:15,783 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check started. >2014-08-25 17:07:15,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:15,919 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:15,949 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:15,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79082988 >2014-08-25 17:07:16,196 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:16,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79082988 >2014-08-25 17:07:16,199 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:07:16,327 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:07:16,328 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:07:16,391 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:16,393 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:16,393 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:16,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: cf5d2ae >2014-08-25 17:07:16,853 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:16,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: cf5d2ae >2014-08-25 17:07:16,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:07:16,894 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check completed. >2014-08-25 17:07:36,895 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-41) Power Management Health Check started. >2014-08-25 17:07:37,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:37,075 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:37,110 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:37,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3cbcee33 >2014-08-25 17:07:37,357 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:37,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3cbcee33 >2014-08-25 17:07:37,360 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:07:37,492 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:07:37,493 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:07:37,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:37,559 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:37,560 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-41) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:37,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5c4227d6 >2014-08-25 17:07:37,798 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:37,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5c4227d6 >2014-08-25 17:07:37,836 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:07:37,837 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-41) Power Management Health Check completed. >2014-08-25 17:07:57,838 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check started. >2014-08-25 17:07:57,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:57,967 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:57,998 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:58,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 29839af8 >2014-08-25 17:07:58,342 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:58,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 29839af8 >2014-08-25 17:07:58,344 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:07:58,473 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:07:58,474 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:07:58,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:07:58,540 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:07:58,541 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:07:58,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 549e6852 >2014-08-25 17:07:58,788 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:07:58,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 549e6852 >2014-08-25 17:07:58,825 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:07:58,826 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check completed. >2014-08-25 17:08:18,827 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check started. >2014-08-25 17:08:18,997 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:08:18,998 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:08:19,030 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:08:19,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7208209a >2014-08-25 17:08:19,565 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:08:19,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7208209a >2014-08-25 17:08:19,567 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:08:19,697 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:08:19,697 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:08:19,827 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:08:19,829 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:08:19,829 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-90) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:08:19,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 57b3192d >2014-08-25 17:08:20,058 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:08:20,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 57b3192d >2014-08-25 17:08:20,093 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:08:20,095 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-90) Power Management Health Check completed. >2014-08-25 17:08:40,099 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check started. >2014-08-25 17:08:40,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:08:40,292 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:08:40,323 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:08:40,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 28b3f6b3 >2014-08-25 17:08:40,547 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:08:40,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 28b3f6b3 >2014-08-25 17:08:40,550 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:08:40,682 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:08:40,683 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:08:40,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:08:40,749 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:08:40,750 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:08:40,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6f0b5a4e >2014-08-25 17:08:41,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:08:41,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [725e9bfe] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6f0b5a4e >2014-08-25 17:08:41,070 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:08:41,071 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-8) [725e9bfe] Power Management Health Check completed. >2014-08-25 17:09:01,073 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 17:09:01,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:01,201 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:01,232 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 195c09c9 >2014-08-25 17:09:01,485 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:01,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 195c09c9 >2014-08-25 17:09:01,487 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:09:01,640 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:09:01,641 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:09:01,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:01,745 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:01,746 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:01,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 8f743c >2014-08-25 17:09:01,969 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:01,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 8f743c >2014-08-25 17:09:02,005 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:09:02,006 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 17:09:22,006 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check started. >2014-08-25 17:09:22,132 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:22,133 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:22,164 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:22,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 60944da3 >2014-08-25 17:09:22,394 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:22,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 60944da3 >2014-08-25 17:09:22,396 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:09:22,523 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:09:22,524 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:09:22,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:22,591 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:22,592 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-80) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:22,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 24f6bf56 >2014-08-25 17:09:22,826 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:22,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 24f6bf56 >2014-08-25 17:09:22,861 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:09:22,862 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-80) Power Management Health Check completed. >2014-08-25 17:09:42,864 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-54) Power Management Health Check started. >2014-08-25 17:09:42,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:42,993 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:43,025 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:43,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 62aed112 >2014-08-25 17:09:43,745 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:43,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 62aed112 >2014-08-25 17:09:43,748 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:09:43,880 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:09:43,881 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:09:43,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:09:43,948 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:09:43,949 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-54) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:09:43,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4c89bb21 >2014-08-25 17:09:44,172 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:09:44,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4c89bb21 >2014-08-25 17:09:44,216 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:09:44,218 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-54) Power Management Health Check completed. >2014-08-25 17:10:04,223 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check started. >2014-08-25 17:10:04,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:04,491 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:04,533 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:04,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 994d9ea >2014-08-25 17:10:04,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:04,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 994d9ea >2014-08-25 17:10:04,832 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:10:05,068 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:10:05,069 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:10:05,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:05,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:05,224 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:05,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3e8723a7 >2014-08-25 17:10:05,514 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:05,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3e8723a7 >2014-08-25 17:10:05,591 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:10:05,593 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-44) Power Management Health Check completed. >2014-08-25 17:10:25,605 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check started. >2014-08-25 17:10:25,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:25,878 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:25,920 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:25,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 283c6d69 >2014-08-25 17:10:26,226 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:26,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 283c6d69 >2014-08-25 17:10:26,229 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:10:26,564 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:10:26,565 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:10:26,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:26,743 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:26,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:26,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6cfc3f4f >2014-08-25 17:10:27,059 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:27,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6cfc3f4f >2014-08-25 17:10:27,128 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:10:27,129 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check completed. >2014-08-25 17:10:47,130 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check started. >2014-08-25 17:10:47,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:47,406 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:47,448 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:47,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f6b9275 >2014-08-25 17:10:47,742 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:47,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f6b9275 >2014-08-25 17:10:47,744 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:10:47,980 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:10:47,981 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:10:48,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:10:48,135 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:10:48,135 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-53) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:10:48,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 626cc1f9 >2014-08-25 17:10:48,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:10:48,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 626cc1f9 >2014-08-25 17:10:48,725 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:10:48,727 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-53) Power Management Health Check completed. >2014-08-25 17:11:08,727 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-55) Power Management Health Check started. >2014-08-25 17:11:08,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:08,863 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:08,895 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:08,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 25d757ff >2014-08-25 17:11:09,210 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:09,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 25d757ff >2014-08-25 17:11:09,212 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:11:09,341 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:11:09,342 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:11:09,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:09,408 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:09,409 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-55) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:09,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: e973cef >2014-08-25 17:11:09,642 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:09,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: e973cef >2014-08-25 17:11:09,678 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:11:09,679 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-55) Power Management Health Check completed. >2014-08-25 17:11:29,677 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check started. >2014-08-25 17:11:29,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:29,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:29,877 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:29,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 77b63aac >2014-08-25 17:11:32,262 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:32,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 77b63aac >2014-08-25 17:11:32,265 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:11:32,391 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:11:32,392 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:11:32,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:32,464 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:32,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-96) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:32,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 14eeeb6e >2014-08-25 17:11:32,716 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:32,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 14eeeb6e >2014-08-25 17:11:32,764 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:11:32,766 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-96) Power Management Health Check completed. >2014-08-25 17:11:52,766 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check started. >2014-08-25 17:11:52,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:52,892 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:52,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:52,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 595d78f6 >2014-08-25 17:11:53,151 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:53,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 595d78f6 >2014-08-25 17:11:53,153 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:11:53,278 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:11:53,279 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:11:53,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:11:53,344 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:11:53,345 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-66) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:11:53,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3ec5d35e >2014-08-25 17:11:53,577 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:11:53,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3ec5d35e >2014-08-25 17:11:53,614 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:11:53,615 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-66) Power Management Health Check completed. >2014-08-25 17:12:13,616 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check started. >2014-08-25 17:12:13,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:13,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:13,774 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:13,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3b307546 >2014-08-25 17:12:14,014 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:14,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3b307546 >2014-08-25 17:12:14,017 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:12:14,147 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:12:14,148 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:12:14,214 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:14,215 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:14,216 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-72) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:14,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 788688f8 >2014-08-25 17:12:14,439 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:14,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 788688f8 >2014-08-25 17:12:14,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:12:14,477 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-72) Power Management Health Check completed. >2014-08-25 17:12:34,478 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check started. >2014-08-25 17:12:34,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:34,613 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:34,644 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:34,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 764b2ea2 >2014-08-25 17:12:34,873 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:34,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 764b2ea2 >2014-08-25 17:12:34,875 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:12:35,005 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:12:35,006 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:12:35,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:35,073 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:35,074 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-39) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:35,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 43eb6075 >2014-08-25 17:12:35,298 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:35,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 43eb6075 >2014-08-25 17:12:35,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:12:35,336 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-39) Power Management Health Check completed. >2014-08-25 17:12:55,336 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check started. >2014-08-25 17:12:55,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:55,464 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:55,496 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:55,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 578866e5 >2014-08-25 17:12:55,722 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:55,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 578866e5 >2014-08-25 17:12:55,725 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:12:55,861 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:12:55,862 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:12:55,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:12:55,927 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:12:55,928 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-95) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:12:55,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5b0c061d >2014-08-25 17:12:56,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:12:56,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5b0c061d >2014-08-25 17:12:56,193 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:12:56,195 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-95) Power Management Health Check completed. >2014-08-25 17:13:16,195 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-77) Power Management Health Check started. >2014-08-25 17:13:16,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:16,322 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:16,355 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:16,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3c9192ce >2014-08-25 17:13:16,579 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:16,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3c9192ce >2014-08-25 17:13:16,581 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:13:16,746 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:13:16,747 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:13:16,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:16,814 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:16,815 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:16,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 32a9cf23 >2014-08-25 17:13:17,038 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:17,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 32a9cf23 >2014-08-25 17:13:17,074 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:13:17,075 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-77) Power Management Health Check completed. >2014-08-25 17:13:37,076 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 17:13:37,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:37,213 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:37,244 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:37,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a112600 >2014-08-25 17:13:37,470 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:37,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a112600 >2014-08-25 17:13:37,472 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:13:37,669 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:13:37,670 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:13:37,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:37,751 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:37,752 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:37,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 370e09e1 >2014-08-25 17:13:37,992 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:37,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 370e09e1 >2014-08-25 17:13:38,050 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:13:38,051 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 17:13:58,051 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 17:13:58,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:58,180 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:58,211 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:58,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3aa6801 >2014-08-25 17:13:58,620 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:58,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3aa6801 >2014-08-25 17:13:58,622 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:13:58,750 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:13:58,752 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:13:58,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:13:58,818 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:13:58,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:13:58,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: fcfe9ae >2014-08-25 17:13:59,043 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:13:59,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: fcfe9ae >2014-08-25 17:13:59,078 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:13:59,080 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 17:14:19,080 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check started. >2014-08-25 17:14:19,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:14:19,206 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:14:19,237 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:14:19,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 70fe3310 >2014-08-25 17:14:19,460 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:14:19,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 70fe3310 >2014-08-25 17:14:19,463 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:14:19,588 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:14:19,589 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:14:19,653 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:14:19,654 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:14:19,655 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:14:19,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5a7e9c71 >2014-08-25 17:14:20,041 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:14:20,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5a7e9c71 >2014-08-25 17:14:20,077 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:14:20,078 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check completed. >2014-08-25 17:14:40,079 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check started. >2014-08-25 17:14:40,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:14:40,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:14:40,317 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:14:40,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c6bc604 >2014-08-25 17:14:40,541 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:14:40,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c6bc604 >2014-08-25 17:14:40,544 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:14:40,669 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:14:40,670 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:14:40,735 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:14:40,736 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:14:40,737 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:14:40,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 22f504c1 >2014-08-25 17:14:40,960 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:14:40,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 22f504c1 >2014-08-25 17:14:41,003 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:14:41,005 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check completed. >2014-08-25 17:15:01,005 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check started. >2014-08-25 17:15:01,134 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:01,136 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:01,167 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:01,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3f3e6a92 >2014-08-25 17:15:01,973 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:01,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3f3e6a92 >2014-08-25 17:15:01,976 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:15:02,105 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:15:02,106 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:15:02,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:02,180 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:02,181 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:02,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6c57a93a >2014-08-25 17:15:02,528 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:02,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6c57a93a >2014-08-25 17:15:02,564 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:15:02,565 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check completed. >2014-08-25 17:15:22,565 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check started. >2014-08-25 17:15:22,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:22,814 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:22,862 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:22,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f0d8e8 >2014-08-25 17:15:23,090 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:23,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f0d8e8 >2014-08-25 17:15:23,092 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:15:23,262 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:15:23,263 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:15:23,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:23,405 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:23,406 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-98) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:23,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5b88c13d >2014-08-25 17:15:23,633 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:23,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5b88c13d >2014-08-25 17:15:23,671 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:15:23,672 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-98) Power Management Health Check completed. >2014-08-25 17:15:43,674 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check started. >2014-08-25 17:15:43,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:43,801 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:43,833 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:43,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 640a85f3 >2014-08-25 17:15:44,060 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:44,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 640a85f3 >2014-08-25 17:15:44,063 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:15:44,187 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:15:44,188 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:15:44,252 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:15:44,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:15:44,254 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-82) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:15:44,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7f641bd4 >2014-08-25 17:15:44,485 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:15:44,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7f641bd4 >2014-08-25 17:15:44,536 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:15:44,537 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-82) Power Management Health Check completed. >2014-08-25 17:16:04,539 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-49) Power Management Health Check started. >2014-08-25 17:16:04,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:04,667 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:04,698 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:04,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6af64871 >2014-08-25 17:16:05,010 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:05,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6af64871 >2014-08-25 17:16:05,013 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:16:05,141 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:16:05,142 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:16:05,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:05,208 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:05,209 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:05,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: d02ab4e >2014-08-25 17:16:05,434 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:05,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: d02ab4e >2014-08-25 17:16:05,475 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:16:05,476 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-49) Power Management Health Check completed. >2014-08-25 17:16:25,476 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check started. >2014-08-25 17:16:25,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:25,608 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:25,639 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:25,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 45ca4e1a >2014-08-25 17:16:25,863 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:25,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 45ca4e1a >2014-08-25 17:16:25,865 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:16:26,017 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:16:26,018 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:16:26,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:26,115 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:26,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:26,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6bf7c002 >2014-08-25 17:16:26,453 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:26,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6bf7c002 >2014-08-25 17:16:26,488 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:16:26,490 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-5) Power Management Health Check completed. >2014-08-25 17:16:46,491 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-79) Power Management Health Check started. >2014-08-25 17:16:46,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:46,616 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:46,647 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:46,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 26e0ee32 >2014-08-25 17:16:46,875 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:46,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 26e0ee32 >2014-08-25 17:16:46,877 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:16:47,004 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:16:47,005 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:16:47,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:16:47,070 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:16:47,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:16:47,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 69cd86c8 >2014-08-25 17:16:47,301 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:16:47,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 69cd86c8 >2014-08-25 17:16:47,336 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:16:47,337 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-79) Power Management Health Check completed. >2014-08-25 17:17:07,338 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-57) Power Management Health Check started. >2014-08-25 17:17:07,465 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:07,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:07,497 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:07,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5af42670 >2014-08-25 17:17:07,724 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:07,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5af42670 >2014-08-25 17:17:07,726 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:17:07,854 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:17:07,855 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:17:07,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:07,922 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:07,923 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-57) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:07,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 34d370ce >2014-08-25 17:17:08,151 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:08,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 34d370ce >2014-08-25 17:17:08,186 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:17:08,187 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-57) Power Management Health Check completed. >2014-08-25 17:17:28,188 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 17:17:28,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:28,339 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:28,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:28,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5ed9f07b >2014-08-25 17:17:28,620 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:28,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5ed9f07b >2014-08-25 17:17:28,623 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:17:28,753 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:17:28,754 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:17:28,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:28,820 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:28,821 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:28,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 42cd8851 >2014-08-25 17:17:29,048 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:29,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 42cd8851 >2014-08-25 17:17:29,084 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:17:29,085 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 17:17:49,087 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 17:17:49,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:49,214 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:49,245 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:49,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 322e465c >2014-08-25 17:17:49,471 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:49,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 322e465c >2014-08-25 17:17:49,473 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:17:49,598 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:17:49,599 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:17:49,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:17:49,668 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:17:49,669 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:17:49,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2e48f2b9 >2014-08-25 17:17:50,067 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:17:50,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2e48f2b9 >2014-08-25 17:17:50,102 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:17:50,104 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 17:18:10,104 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check started. >2014-08-25 17:18:10,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:10,270 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:10,301 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:10,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f8e30ff >2014-08-25 17:18:10,528 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:10,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f8e30ff >2014-08-25 17:18:10,531 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:18:10,683 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:18:10,684 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:18:10,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:10,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:10,754 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:10,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 59541a33 >2014-08-25 17:18:10,977 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:10,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 59541a33 >2014-08-25 17:18:11,046 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:18:11,047 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check completed. >2014-08-25 17:18:31,048 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check started. >2014-08-25 17:18:31,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:31,179 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:31,211 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:31,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 423f2d23 >2014-08-25 17:18:31,436 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:31,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 423f2d23 >2014-08-25 17:18:31,438 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:18:31,565 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:18:31,565 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:18:31,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:31,663 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:31,663 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:31,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 72233b72 >2014-08-25 17:18:31,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:31,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 72233b72 >2014-08-25 17:18:31,923 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:18:31,925 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check completed. >2014-08-25 17:18:51,926 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-100) Power Management Health Check started. >2014-08-25 17:18:52,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:52,059 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:52,090 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:52,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 35029107 >2014-08-25 17:18:52,313 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:52,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 35029107 >2014-08-25 17:18:52,315 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:18:52,445 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:18:52,445 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:18:52,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:18:52,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:18:52,512 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-100) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:18:52,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1777b192 >2014-08-25 17:18:52,740 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:18:52,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1777b192 >2014-08-25 17:18:52,776 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:18:52,777 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-100) Power Management Health Check completed. >2014-08-25 17:19:12,778 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check started. >2014-08-25 17:19:12,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:12,904 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:12,935 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:12,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5bd550f0 >2014-08-25 17:19:13,165 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:13,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5bd550f0 >2014-08-25 17:19:13,168 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:19:13,300 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:19:13,300 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:19:13,364 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:13,365 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:13,366 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:13,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 49213969 >2014-08-25 17:19:13,631 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:13,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 49213969 >2014-08-25 17:19:13,666 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:19:13,667 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check completed. >2014-08-25 17:19:33,669 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check started. >2014-08-25 17:19:33,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:33,797 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:33,828 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:33,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 33b8108d >2014-08-25 17:19:34,061 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:34,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 33b8108d >2014-08-25 17:19:34,063 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:19:34,216 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:19:34,217 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:19:34,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:34,282 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:34,283 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-29) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:34,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 41a06a3 >2014-08-25 17:19:34,703 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:34,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 41a06a3 >2014-08-25 17:19:34,740 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:19:34,742 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-29) Power Management Health Check completed. >2014-08-25 17:19:54,742 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check started. >2014-08-25 17:19:54,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:54,872 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:54,905 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:54,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 43c3cd1a >2014-08-25 17:19:55,130 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:55,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 43c3cd1a >2014-08-25 17:19:55,132 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:19:55,265 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:19:55,266 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:19:55,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:19:55,357 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:19:55,358 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-20) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:19:55,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1c8bff5a >2014-08-25 17:19:55,590 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:19:55,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1c8bff5a >2014-08-25 17:19:55,626 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:19:55,628 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-20) Power Management Health Check completed. >2014-08-25 17:20:15,628 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check started. >2014-08-25 17:20:15,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:15,757 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:15,788 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:15,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6e060ff1 >2014-08-25 17:20:16,679 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:16,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6e060ff1 >2014-08-25 17:20:16,681 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:20:16,808 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:20:16,809 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:20:16,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:16,953 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:16,953 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:16,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 51ce9f31 >2014-08-25 17:20:17,183 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:17,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 51ce9f31 >2014-08-25 17:20:17,219 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:20:17,220 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-78) Power Management Health Check completed. >2014-08-25 17:20:37,222 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check started. >2014-08-25 17:20:37,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:37,351 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:37,383 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:37,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4c9867dd >2014-08-25 17:20:37,613 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:37,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4c9867dd >2014-08-25 17:20:37,616 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:20:37,744 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:20:37,745 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:20:37,810 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:37,811 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:37,812 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:37,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 71887e1b >2014-08-25 17:20:38,046 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:38,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 71887e1b >2014-08-25 17:20:38,082 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:20:38,083 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check completed. >2014-08-25 17:20:58,084 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check started. >2014-08-25 17:20:58,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:58,215 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:58,246 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:58,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3737f8cd >2014-08-25 17:20:58,565 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:58,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3737f8cd >2014-08-25 17:20:58,567 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:20:58,695 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:20:58,696 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:20:58,762 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:20:58,763 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:20:58,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-13) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:20:58,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4f41dfef >2014-08-25 17:20:58,991 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:20:58,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4f41dfef >2014-08-25 17:20:59,027 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:20:59,028 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-13) Power Management Health Check completed. >2014-08-25 17:21:19,029 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check started. >2014-08-25 17:21:19,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:21:19,156 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:21:19,187 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:21:19,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2a392f87 >2014-08-25 17:21:19,412 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:21:19,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2a392f87 >2014-08-25 17:21:19,415 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:21:19,542 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:21:19,542 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:21:19,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:21:19,648 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:21:19,649 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-93) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:21:19,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3979fe12 >2014-08-25 17:21:20,016 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:21:20,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3979fe12 >2014-08-25 17:21:20,052 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:21:20,053 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-93) Power Management Health Check completed. >2014-08-25 17:21:40,054 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check started. >2014-08-25 17:21:40,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:21:40,185 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:21:40,221 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:21:40,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 61c9e2d >2014-08-25 17:21:40,519 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:21:40,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 61c9e2d >2014-08-25 17:21:40,521 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:21:40,648 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:21:40,649 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:21:40,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:21:40,716 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:21:40,717 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-65) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:21:40,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: cc2d341 >2014-08-25 17:21:40,944 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:21:40,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: cc2d341 >2014-08-25 17:21:40,979 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:21:40,980 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-65) Power Management Health Check completed. >2014-08-25 17:22:00,981 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-45) Power Management Health Check started. >2014-08-25 17:22:01,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:01,110 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:01,141 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:01,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 79b03f6f >2014-08-25 17:22:01,371 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:01,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 79b03f6f >2014-08-25 17:22:01,373 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:22:01,500 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:22:01,501 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:22:01,567 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:01,568 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:01,569 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:01,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6c49cae >2014-08-25 17:22:01,801 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:01,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6c49cae >2014-08-25 17:22:01,836 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:22:01,838 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-45) Power Management Health Check completed. >2014-08-25 17:22:21,838 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check started. >2014-08-25 17:22:21,964 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:21,966 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:22,001 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:22,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 745c30ff >2014-08-25 17:22:22,237 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:22,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 745c30ff >2014-08-25 17:22:22,240 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:22:22,369 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:22:22,370 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:22:22,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:22,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:22,440 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:22,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2e5a4989 >2014-08-25 17:22:22,736 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:22,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2e5a4989 >2014-08-25 17:22:22,815 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:22:22,817 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check completed. >2014-08-25 17:22:42,818 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-92) Power Management Health Check started. >2014-08-25 17:22:42,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:42,948 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:42,980 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:42,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 66524a69 >2014-08-25 17:22:43,231 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:43,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 66524a69 >2014-08-25 17:22:43,233 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:22:43,379 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:22:43,380 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:22:43,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:22:43,449 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:22:43,450 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-92) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:22:43,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 76cdc3b6 >2014-08-25 17:22:43,695 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:22:43,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 76cdc3b6 >2014-08-25 17:22:43,733 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:22:43,734 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-92) Power Management Health Check completed. >2014-08-25 17:23:03,736 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check started. >2014-08-25 17:23:03,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:03,888 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:03,920 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:03,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 66460a17 >2014-08-25 17:23:04,174 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:04,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 66460a17 >2014-08-25 17:23:04,176 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:23:04,375 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:23:04,376 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:23:04,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:04,465 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:04,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:04,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 17ef302b >2014-08-25 17:23:04,859 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:04,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 17ef302b >2014-08-25 17:23:04,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:23:04,896 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-50) Power Management Health Check completed. >2014-08-25 17:23:24,896 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check started. >2014-08-25 17:23:25,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:25,033 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:25,065 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:25,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 4670ff5e >2014-08-25 17:23:25,291 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:25,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 4670ff5e >2014-08-25 17:23:25,293 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:23:25,457 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:23:25,457 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:23:25,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:25,535 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:25,536 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-18) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:25,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 74a724b3 >2014-08-25 17:23:25,768 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:25,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 74a724b3 >2014-08-25 17:23:25,804 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:23:25,805 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-18) Power Management Health Check completed. >2014-08-25 17:23:45,806 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check started. >2014-08-25 17:23:45,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:45,935 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:45,967 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:45,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3a16fe39 >2014-08-25 17:23:46,196 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:46,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3a16fe39 >2014-08-25 17:23:46,199 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:23:46,336 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:23:46,337 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:23:46,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:23:46,510 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:23:46,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-15) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:23:46,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3135fbf7 >2014-08-25 17:23:46,754 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:23:46,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3135fbf7 >2014-08-25 17:23:46,791 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:23:46,793 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-15) Power Management Health Check completed. >2014-08-25 17:24:06,793 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check started. >2014-08-25 17:24:06,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:06,922 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:06,954 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:06,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c46c597 >2014-08-25 17:24:07,305 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:07,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c46c597 >2014-08-25 17:24:07,308 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:24:07,438 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:24:07,439 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:24:07,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:07,514 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:07,515 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-68) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:07,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1617426b >2014-08-25 17:24:07,781 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:07,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1617426b >2014-08-25 17:24:07,818 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:24:07,819 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-68) Power Management Health Check completed. >2014-08-25 17:24:27,821 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-35) Power Management Health Check started. >2014-08-25 17:24:27,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:27,951 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:27,983 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:27,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 110f32d0 >2014-08-25 17:24:28,216 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:28,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 110f32d0 >2014-08-25 17:24:28,219 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:24:28,349 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:24:28,350 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:24:28,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:28,418 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:28,419 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-35) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:28,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1e1da071 >2014-08-25 17:24:28,994 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:28,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1e1da071 >2014-08-25 17:24:29,045 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:24:29,046 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-35) Power Management Health Check completed. >2014-08-25 17:24:49,047 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check started. >2014-08-25 17:24:49,175 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:49,176 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:49,208 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:49,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6cce9ba9 >2014-08-25 17:24:49,700 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:49,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6cce9ba9 >2014-08-25 17:24:49,702 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:24:49,845 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:24:49,846 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:24:49,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:24:49,922 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:24:49,923 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-99) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:24:49,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1a01059c >2014-08-25 17:24:50,150 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:24:50,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1a01059c >2014-08-25 17:24:50,212 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:24:50,213 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-99) Power Management Health Check completed. >2014-08-25 17:25:10,215 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check started. >2014-08-25 17:25:10,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:10,342 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:10,375 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:10,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6ee78afb >2014-08-25 17:25:10,601 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:10,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6ee78afb >2014-08-25 17:25:10,603 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:25:10,730 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:25:10,731 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:25:10,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:10,800 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:10,801 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:10,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 28b06ce6 >2014-08-25 17:25:11,024 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:11,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 28b06ce6 >2014-08-25 17:25:11,060 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:25:11,061 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-76) Power Management Health Check completed. >2014-08-25 17:25:31,062 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-45) Power Management Health Check started. >2014-08-25 17:25:31,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:31,190 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:31,222 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:31,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1b0d8463 >2014-08-25 17:25:31,942 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:31,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1b0d8463 >2014-08-25 17:25:31,944 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:25:32,113 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:25:32,113 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:25:32,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:32,185 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:32,186 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-45) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:32,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2c51fcc9 >2014-08-25 17:25:32,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:32,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2c51fcc9 >2014-08-25 17:25:32,447 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:25:32,448 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-45) Power Management Health Check completed. >2014-08-25 17:25:52,448 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check started. >2014-08-25 17:25:52,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:52,578 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:52,610 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:52,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 33f7b675 >2014-08-25 17:25:52,840 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:52,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 33f7b675 >2014-08-25 17:25:52,843 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:25:52,968 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:25:52,969 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:25:53,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:25:53,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:25:53,036 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-7) [32a9def7] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:25:53,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6e1610b6 >2014-08-25 17:25:53,270 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:25:53,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [32a9def7] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6e1610b6 >2014-08-25 17:25:53,305 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [32a9def7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:25:53,306 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-7) [32a9def7] Power Management Health Check completed. >2014-08-25 17:26:13,307 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check started. >2014-08-25 17:26:13,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:13,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:13,497 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:13,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 73a3c1ab >2014-08-25 17:26:13,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:13,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 73a3c1ab >2014-08-25 17:26:13,799 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:26:13,924 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:26:13,925 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:26:13,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:13,993 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:13,994 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-88) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:13,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1133029 >2014-08-25 17:26:14,229 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:14,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1133029 >2014-08-25 17:26:14,265 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:26:14,267 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-88) Power Management Health Check completed. >2014-08-25 17:26:34,267 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check started. >2014-08-25 17:26:34,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:34,398 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:34,429 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:34,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 5a2fb32c >2014-08-25 17:26:34,666 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:34,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 5a2fb32c >2014-08-25 17:26:34,669 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:26:34,798 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:26:34,799 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:26:35,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:35,042 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:35,043 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-60) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:35,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 2ecbfd40 >2014-08-25 17:26:35,285 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:35,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 2ecbfd40 >2014-08-25 17:26:35,321 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:26:35,322 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-60) Power Management Health Check completed. >2014-08-25 17:26:55,322 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check started. >2014-08-25 17:26:55,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:55,449 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:55,480 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:55,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 756170d6 >2014-08-25 17:26:55,714 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:55,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 756170d6 >2014-08-25 17:26:55,717 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:26:55,846 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:26:55,847 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:26:55,913 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:26:55,914 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:26:55,915 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-16) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:26:55,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 53e4fb29 >2014-08-25 17:26:56,140 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:26:56,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 53e4fb29 >2014-08-25 17:26:56,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:26:56,177 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-16) Power Management Health Check completed. >2014-08-25 17:27:16,177 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-14) Power Management Health Check started. >2014-08-25 17:27:16,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:16,499 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:16,534 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:16,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 3f57e2f9 >2014-08-25 17:27:16,786 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:16,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 3f57e2f9 >2014-08-25 17:27:16,788 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:27:16,921 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:27:16,922 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:27:17,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:17,157 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:17,158 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-14) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:17,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 1deefb83 >2014-08-25 17:27:17,396 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:17,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 1deefb83 >2014-08-25 17:27:17,432 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:27:17,433 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-14) Power Management Health Check completed. >2014-08-25 17:27:37,434 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check started. >2014-08-25 17:27:37,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:37,577 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:37,609 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:37,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 7941339b >2014-08-25 17:27:37,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:37,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 7941339b >2014-08-25 17:27:37,891 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:27:38,019 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:27:38,020 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:27:38,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:38,089 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:38,090 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:38,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 6a8c58fb >2014-08-25 17:27:38,323 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:38,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 6a8c58fb >2014-08-25 17:27:38,364 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:27:38,365 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-84) Power Management Health Check completed. >2014-08-25 17:27:58,367 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check started. >2014-08-25 17:27:58,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:58,494 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:58,527 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:58,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 21b7f919 >2014-08-25 17:27:58,761 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:58,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 21b7f919 >2014-08-25 17:27:58,763 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:27:58,889 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:27:58,890 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:27:58,996 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:27:58,997 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:27:58,998 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-22) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:27:59,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 16f883ca >2014-08-25 17:27:59,232 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:27:59,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 16f883ca >2014-08-25 17:27:59,269 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:27:59,271 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-22) Power Management Health Check completed. >2014-08-25 17:28:19,271 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check started. >2014-08-25 17:28:19,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:28:19,437 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:28:19,471 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:28:19,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: 25ac5140 >2014-08-25 17:28:19,784 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:28:19,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: 25ac5140 >2014-08-25 17:28:19,786 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-25 17:28:19,934 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-25 17:28:19,935 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-25 17:28:20,000 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-25 17:28:20,001 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-25 17:28:20,002 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [21537b5b] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:rootfdh, Options:, Fencing policy:null >2014-08-25 17:28:20,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = rootfdh, password = ******, options = '', policy = 'null'), log id: eacb09b >2014-08-25 17:28:20,386 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed] >2014-08-25 17:28:20,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [21537b5b] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed], log id: eacb09b >2014-08-25 17:28:20,422 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [21537b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Health check failed on Host rose07.qa.lab.tlv.redhat.com primary sequential agent, future fence operations may fail is secondary agent if not defined properly. >2014-08-25 17:28:20,424 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (DefaultQuartzScheduler_Worker-9) [21537b5b] Power Management Health Check completed. >2014-08-25 17:28:51,795 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-25 17:28:51,846 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-25 17:28:51,847 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-25 17:28:51,848 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-25 17:28:51,849 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-25 17:28:51,850 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-25 17:28:51,852 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-25 17:28:51,853 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-25 17:28:51,853 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-25 17:28:51,854 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-25 17:28:51,855 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-25 17:28:51,856 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-25 17:28:51,876 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-25 17:28:51,877 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-25 17:28:51,878 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-25 17:28:51,878 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-25 17:28:51,880 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-25 17:28:51,881 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-25 17:28:51,881 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-25 17:28:51,882 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-25 17:28:51,883 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-25 17:28:51,883 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-25 17:28:51,884 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-25 17:28:51,885 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-25 17:28:51,886 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-25 17:28:51,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-25 17:28:51,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-25 17:28:51,900 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-25 17:28:51,901 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-25 17:28:51,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-25 17:28:51,902 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-25 17:28:51,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-25 17:28:51,903 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-25 17:28:51,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-25 17:28:51,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-25 17:28:51,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-25 17:28:51,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-25 17:28:51,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-25 17:28:51,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-25 17:28:51,929 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-25 17:28:51,929 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-25 17:28:51,930 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-25 17:28:51,931 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-25 17:28:51,931 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-25 17:28:51,932 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-25 17:28:51,933 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-25 17:28:51,933 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-25 17:28:51,934 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-25 17:28:51,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-25 17:28:51,936 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-25 17:28:51,936 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-25 17:28:51,937 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-25 17:28:51,938 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-25 17:28:51,939 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-25 17:28:51,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-25 17:28:51,954 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-25 17:28:51,954 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-25 17:28:51,955 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-25 17:28:51,956 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-25 17:28:51,956 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-25 17:28:51,957 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-25 17:28:51,958 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-25 17:28:51,958 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-25 17:28:51,959 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-25 17:28:51,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-25 17:28:51,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-25 17:28:51,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-25 17:28:51,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-25 17:28:54,171 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-25 17:28:55,520 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-25 17:28:55,522 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-25 17:28:55,890 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140821064931.gitb794d66.el6 >2014-08-25 17:28:55,891 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-25 17:28:55,898 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-25 17:28:55,899 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-25 17:28:55,922 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-25 17:28:56,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-25 17:28:56,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-25 17:28:56,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-25 17:28:56,016 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-25 17:28:56,019 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-25 17:28:56,045 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-25 17:28:56,072 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-25 17:28:56,073 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-25 17:28:56,082 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-25 17:28:56,354 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-25 17:28:56,442 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-25 17:28:56,492 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-25 17:28:56,494 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-25 17:28:56,495 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-25 17:28:56,496 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-25 17:28:56,496 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-25 17:28:56,624 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-25 17:28:56,625 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-25 17:28:56,626 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-25 17:28:56,628 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-25 17:28:56,631 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-25 17:28:56,638 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-25 17:28:56,766 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-25 17:28:56,767 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 17:28:56,785 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-25 17:28:56,786 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-25 17:28:57,171 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-1) Start initializing PmHealthCheckManager >2014-08-25 17:28:57,172 INFO [org.ovirt.engine.core.bll.pm.PmHealthCheckManager] (MSC service thread 1-1) Finished initializing PmHealthCheckManager >2014-08-25 17:28:57,174 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-25 17:28:57,186 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-25 17:28:57,213 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-25 17:28:57,215 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-25 17:28:57,567 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 17:28:57,655 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-25 17:28:57,867 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-25 17:28:58,112 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-25 17:28:58,112 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 17:28:58,141 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-25 17:28:58,166 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-25 17:28:58,167 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-25 17:28:58,196 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-25 17:28:58,223 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-25 17:28:58,283 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-25 17:28:58,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-25 17:28:58,304 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-25 17:28:58,305 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-25 17:28:58,306 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-25 17:28:58,306 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-25 17:28:58,356 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-25 17:28:58,357 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-25 17:28:58,363 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-25 17:28:58,407 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-25 17:28:58,440 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-25 17:28:58,467 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-25 17:28:58,467 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-25 17:28:58,468 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-25 17:28:58,470 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-25 17:29:03,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 67f4664b >2014-08-25 17:29:03,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 67f4664b >2014-08-25 17:29:03,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 762bc0e >2014-08-25 17:29:03,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 6775de36 >2014-08-25 17:29:03,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 6775de36 >2014-08-25 17:29:03,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 762bc0e >2014-08-25 17:29:04,611 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-25 17:29:04,609 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-25 17:29:04,620 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-25 17:29:04,956 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [6ea74b2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-25 17:29:04,956 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [2938b01c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-25 17:29:04,997 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [63d11b9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-25 17:29:08,596 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-25 17:29:08,651 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-25 17:29:08,699 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-25 17:29:08,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: UNASSIGNED not exist in string table >2014-08-25 17:29:08,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-25 17:29:08,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-25 17:29:08,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-25 17:29:08,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-25 17:29:08,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-25 17:29:08,761 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-25 17:29:08,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-25 17:29:08,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-25 17:29:08,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-25 17:29:08,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: DWH_STOPPED not exist in string table >2014-08-25 17:29:08,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: DWH_STARTED not exist in string table >2014-08-25 17:29:08,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: DWH_ERROR not exist in string table >2014-08-25 17:29:08,793 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-25 17:29:08,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-25 17:29:08,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-25 17:29:08,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-25 17:29:09,288 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Correlation ID: 6f2facbf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-25 17:29:09,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [6f2facbf] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-25 17:29:09,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-25 17:29:09,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [6f2facbf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-25 17:29:09,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 4ca314bd >2014-08-25 17:29:09,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-25 17:29:10,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 4c0be942 >2014-08-25 17:29:10,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4c0be942 >2014-08-25 17:29:10,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ca314bd >2014-08-25 17:29:10,052 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [6f2facbf] Discovered no tasks on Storage Pool DC1 >2014-08-25 17:29:29,625 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [7ed72538] Running command: LogoutUserCommand internal: false. >2014-08-25 17:29:29,631 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [68dc9ec6] Running command: LogoutUserCommand internal: false. >2014-08-25 17:29:29,636 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [1ec85c05] Running command: LogoutUserCommand internal: false. >2014-08-25 17:29:29,620 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3e5001a3] Running command: LogoutUserCommand internal: false. >2014-08-25 17:29:29,687 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [68dc9ec6] Correlation ID: 68dc9ec6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-25 17:29:29,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3e5001a3] Correlation ID: 3e5001a3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-25 17:29:29,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [1ec85c05] Correlation ID: 1ec85c05, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-25 17:29:29,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [7ed72538] Correlation ID: 7ed72538, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1133611
: 930541